Nov 03 13:05:46.057380 ip-10-0-132-167 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 13:05:46.057391 ip-10-0-132-167 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 13:05:46.057399 ip-10-0-132-167 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 13:05:46.057744 ip-10-0-132-167 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 13:05:56.532674 ip-10-0-132-167 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Nov 03 13:05:56.532690 ip-10-0-132-167 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b1f81186ba524625b5bed04a097307e7 --
Nov 03 13:07:52.144616 ip-10-0-132-167 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 13:07:52.532118 ip-10-0-132-167 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:07:52.532118 ip-10-0-132-167 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 13:07:52.532118 ip-10-0-132-167 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:07:52.532118 ip-10-0-132-167 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:07:52.538812 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538724    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:07:52.538812 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538809    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:07:52.538812 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538812    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:07:52.538812 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538816    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:07:52.538812 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538819    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538822    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538826    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538829    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538832    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538834    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538837    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538840    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538843    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538846    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538848    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538851    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538853    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538856    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538859    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538861    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538864    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538866    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538869    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:07:52.539002 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538879    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538882    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538884    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538887    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538890    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538892    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538895    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538897    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538900    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538902    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538905    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538907    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538910    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538913    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538916    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538919    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538922    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538925    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538927    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:07:52.539466 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538930    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538932    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538935    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538938    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538940    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538943    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538945    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538948    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538950    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538955    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538959    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538967    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538970    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538972    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538975    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538978    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538981    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538983    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538985    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:07:52.539910 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538988    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538991    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538993    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538996    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.538999    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539004    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539007    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539012    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539015    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539018    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539021    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539023    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539026    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539029    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539031    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539034    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539036    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539039    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539042    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539044    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:07:52.540384 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539047    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539049    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539053    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539056    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539058    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539061    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539063    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539066    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539068    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539071    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539073    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539076    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539079    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539081    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539084    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539087    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539089    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539093    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539096    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539098    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:07:52.540855 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539102    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539104    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539107    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539109    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.539112    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540775    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540782    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540786    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540789    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540792    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540796    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540799    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540802    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540805    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540808    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540812    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540815    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540818    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540821    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:07:52.541324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540823    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540826    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540829    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540832    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540834    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540837    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540840    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540842    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540845    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540848    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540852    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540856    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540860    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540864    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540867    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540870    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540872    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540875    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540878    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:07:52.541814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540880    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540884    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540888    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540891    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540893    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540896    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540899    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540901    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540905    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540907    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540911    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540914    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540916    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540919    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540921    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540924    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540926    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540929    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540932    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:07:52.542261 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540934    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540937    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540940    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540942    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540945    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540948    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540950    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540952    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540955    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540957    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540960    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540963    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540966    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540969    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540971    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540974    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540976    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540979    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540981    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540984    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:07:52.542738 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540986    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540989    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540992    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540995    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.540998    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541001    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541003    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541006    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541009    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541011    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541014    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541016    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541019    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541021    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541024    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541027    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541029    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541032    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541034    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541037    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:07:52.543211 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541039    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541042    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541044    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541047    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541049    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541052    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541054    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541057    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541059    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541061    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541064    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541066    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541069    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541071    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541561    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541567    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541570    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541573    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541576    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541578    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:07:52.543701 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541581    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541584    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541586    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541589    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541591    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541594    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541597    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541599    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541602    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541605    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541608    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541610    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541613    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541616    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541620    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541623    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541626    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541628    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541631    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:07:52.544696 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541634    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541637    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541639    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541642    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541645    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541647    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541650    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541653    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541655    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541658    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541661    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541664    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541666    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541669    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541671    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541674    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541676    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541680    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541682    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:07:52.545142 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541685    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541689    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541693    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541697    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541700    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541703    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541706    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541709    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541711    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541714    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541716    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541719    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541721    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541724    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541726    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541729    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541731    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541734    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541737    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541739    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:07:52.545603 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541742    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541745    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541748    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541750    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541753    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541756    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541758    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541761    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541763    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541766    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541768    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541770    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541776    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541779    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541781    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541784    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541786    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541789    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541791    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:07:52.546086 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541794    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541796    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541799    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541801    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541804    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541806    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541809    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541812    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541814    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541817    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541819    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541822    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541825    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541828    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541830    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541833    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541835    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541838    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541841    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541844    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:07:52.546585 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541846    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541848    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.541851    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542801    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542812    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542820    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542825    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542830    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542833    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542840    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542844    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542848    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542851    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542854    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542858    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542861    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542864    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542867    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542870    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542873    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542876    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542881    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542883    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 13:07:52.547043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542887    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542892    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542895    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542904    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542907    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542911    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542914    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542917    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542921    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542925    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542928    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542931    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542935    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542938    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542941    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542944    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542947    2577 flags.go:64] FLAG: --enable-server="true"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542950    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542954    2577 flags.go:64] FLAG: --event-burst="100"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542957    2577 flags.go:64] FLAG: --event-qps="50"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542961    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542963    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542966    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542970    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 13:07:52.547586 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542974    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542976    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542980    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542983    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542986    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542989    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542992    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542995    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.542997    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543002    2577 flags.go:64] FLAG: --feature-gates=""
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543006    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543009    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543012    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543015    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543018    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543022    2577 flags.go:64] FLAG: --help="false"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543025    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543029    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543032    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543035    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543039    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543042    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543045    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 13:07:52.548172 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543048    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543051    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543054    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543057    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543060    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543063    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543066    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543069    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543072    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543075    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543078    2577 flags.go:64] FLAG: --lock-file=""
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543081    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543084    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543087    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543092    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543095    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543098    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543101    2577 flags.go:64] FLAG: --logging-format="text"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543106    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543109    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543112    2577 flags.go:64] FLAG: --manifest-url=""
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543115    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543119    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543122    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543126    2577 flags.go:64] FLAG: --max-pods="110"
Nov 03 13:07:52.548832 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543130    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543133    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543136    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543139    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543142    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543145    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543148    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543157    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543160    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543163    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543166    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543169    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543172    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543175    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543178    2577 flags.go:64] FLAG: --port="10250"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543181    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543184    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-062da26ab9667c414"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543187    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543189    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543193    2577 flags.go:64] FLAG: --register-node="true"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543196    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543199    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543202    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543205    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543208    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543213    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 13:07:52.549463 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543217    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543220    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543223    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543226    2577 flags.go:64] FLAG: --runonce="false"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543229    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543232    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543237    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543240    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543243    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543247    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543250    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543253    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543273    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543277    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543281    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543284    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543288    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543291    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543294    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543300    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543303    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543306    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543310    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543313    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543316    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 13:07:52.550181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543319    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543321    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543324    2577 flags.go:64] FLAG: --v="2"
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543329    2577 flags.go:64] FLAG: --version="false"
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543334    2577 flags.go:64] FLAG: --vmodule=""
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543337    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543352    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543492    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543496    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543499    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543503    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543505    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543508    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543513    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543516    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543518    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543521    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543524    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543527    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543529    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543532    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:07:52.550795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543535    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543537    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543540    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543543    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543545    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543548    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543551    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543553    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543556    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543558    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543561    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543563    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543566    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543568    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543571    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543574    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543576    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543579    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543582    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543585    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:07:52.551292 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543587    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543590    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543592    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543595    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543599    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543602    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543604    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543607    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543609    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543613    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543615    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543619    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543623    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543627    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543629    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543632    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543636    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543638    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543641    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543643    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:07:52.551876 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543646    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543649    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543651    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543654    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543656    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543658    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543661    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543664    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543666    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543669    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543672    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543675    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543677    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543680    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543682    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543685    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543689    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543692    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543694    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:07:52.552372 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543697    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543699    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543702    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543705    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543708    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543712    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543714    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543717    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543720    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543722    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543725    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543728    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543730    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543733    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543735    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543738    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543740    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543743    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543745    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:07:52.552837 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543749    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543752    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543755    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543758    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543761    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543764    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543766    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543769    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543772    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543774    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543778    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543781    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543784    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543786    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.543812    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543852    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:07:52.553289 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543856    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543860    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543865    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543868    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543870    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543873    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543876    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543878    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543881    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543883    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543886    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543901    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543904    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543907    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543910    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543913    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543916    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543918    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543921    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543924    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:07:52.553685 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543926    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543929    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543932    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543935    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543937    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543940    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543942    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543946    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543949    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543952    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543955    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543957    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543960    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543962    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543967    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543970    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543973    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543976    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543978    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543981    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:07:52.554147 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543984    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543986    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543989    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543992    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543995    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.543998    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544000    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544003    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544005    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544008    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544010    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544013    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544016    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544018    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544022    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544026    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544029    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544031    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544034    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:07:52.554629 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544038    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544041    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544043    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544046    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544048    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544052    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544054    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544058    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544061    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544064    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544066    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544069    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544071    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544074    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544077    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544079    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544082    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544091    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544096    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544098    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:07:52.555075 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544101    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544104    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544106    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544109    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544111    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544114    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544116    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544118    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544121    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544124    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544127    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544129    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544133    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544136    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544138    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544141    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544143    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544146    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544148    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544153    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:07:52.555564 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544155    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544159    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544164    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544167    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544171    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.544173    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.551371    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.551387    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.552014    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.552020    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552075    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552080    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552092    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552096    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552099    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552102    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552105    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552107    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552110    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:07:52.556051 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552113    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552116    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552118    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552121    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552124    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552127    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552130    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552132    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552135    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552137    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552140    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552142    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552145    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552148    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552150    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552153    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552155    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552158    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552160    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:07:52.556543 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552163    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552167    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552170    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552172    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552175    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552177    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552180    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552184    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552188    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552191    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552195    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552198    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552201    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552203    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552206    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552208    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552210    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552213    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552216    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:07:52.556981 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552219    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552221    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552224    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552227    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552229    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552232    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552234    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552237    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552240    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552242    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552245    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552247    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552250    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552253    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552255    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552258    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552261    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552264    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552266    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552269    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:07:52.557445 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552271    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552274    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552276    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552279    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552281    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552283    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552286    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552288    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552291    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552293    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552296    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552298    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552301    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552304    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552307    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552309    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552312    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552315    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552317    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:07:52.557904 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552320    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552322    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552325    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552327    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552330    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552332    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552335    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552338    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552352    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552356    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552358    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552361    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552364    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552367    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552369    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552372    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552375    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552377    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552380    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:07:52.558369 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552383    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.552410    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552575    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552580    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552583    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552586    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552588    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552591    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552593    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552596    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552599    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552601    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552604    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552607    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552609    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552612    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:07:52.558814 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552614    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552617    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552619    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552622    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552624    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552627    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552629    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552632    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552635    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552638    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552640    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552643    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552646    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552648    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552650    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552653    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552655    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552658    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552660    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552663    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:07:52.559195 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552665    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552667    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552670    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552672    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552675    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552677    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552680    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552682    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552685    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552688    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552690    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552693    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552695    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552698    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552700    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552703    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552705    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552708    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552711    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552713    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:07:52.559818 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552716    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552719    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552721    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552723    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552726    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552729    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552731    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552733    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552736    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552738    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552742    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552746    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552749    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552752    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552755    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552757    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552760    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552762    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552765    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:07:52.560380 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552768    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552770    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552772    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552776    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552778    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552780    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552783    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552785    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552788    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552790    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552793    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552796    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552798    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552801    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552803    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552806    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552808    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552811    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552815    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:07:52.561067 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552818    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552821    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552824    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552827    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552829    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552832    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552834    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552837    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552840    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552843    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552845    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552848    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552851    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552854    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.552876    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552906    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:07:52.561795 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552910    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552913    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552915    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552918    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552921    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552923    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552926    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552928    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552931    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552933    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552936    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552938    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552941    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552944    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552946    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552948    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552951    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552953    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552956    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552958    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:07:52.562651 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552961    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552963    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552966    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552968    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552971    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552973    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552975    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552979    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552983    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552986    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552988    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552991    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552994    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.552996    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553002    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553005    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553007    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553010    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553012    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:07:52.563131 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553015    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553017    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553020    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553022    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553025    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553028    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553031    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553034    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553036    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553039    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553041    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553044    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553046    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553049    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553051    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553055    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553059    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553062    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553065    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:07:52.563656 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553068    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553071    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553073    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553076    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553079    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553081    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553084    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553086    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553090    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553093    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553096    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553098    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553101    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553104    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553106    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553109    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553111    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553114    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553116    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553119    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:07:52.564099 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553122    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553125    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553128    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553130    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553132    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553135    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553137    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553140    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553142    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553145    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553147    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553150    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553152    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553155    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553158    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553160    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553164    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553166    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553169    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:07:52.564608 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553171    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553175    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553178    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553180    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553183    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553185    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553188    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:52.553190    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.553241    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.555092    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.556126    2577 server.go:1014] "Starting client certificate rotation"
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.556222    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.556253    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.558144    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 13:07:52.565074 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.562609    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 13:07:52.573759 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.573742    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 03 13:07:52.576838 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.576823    2577 log.go:25] "Validated CRI v1 image API"
Nov 03 13:07:52.577722 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.577698    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 13:07:52.581093 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.581075    2577 fs.go:135] Filesystem UUIDs: map[570e1c45-0b29-4daf-b785-3acf9c15b081:/dev/nvme0n1p4 7483819c-7238-4cde-a9cf-dd7d874e43a0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 03 13:07:52.581168 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.581093    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 13:07:52.586410 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.586279    2577 manager.go:217] Machine: {Timestamp:2026-11-03 13:07:52.584993928 +0000 UTC m=+0.327860394 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101602 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec235da851e4140fee3d94aff9926027 SystemUUID:ec235da8-51e4-140f-ee3d-94aff9926027 BootID:b1f81186-ba52-4625-b5be-d04a097307e7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:ac:87:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:ac:87:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:2d:e8:aa:7f:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 13:07:52.586410 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.586406    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 13:07:52.586517 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.586481    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 13:07:52.589725 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.589682    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 13:07:52.589798 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.589782    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 13:07:52.589896 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.589730    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-167.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 13:07:52.589938 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.589900    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 13:07:52.589938 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.589909    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 13:07:52.589993 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.589948    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 13:07:52.589993 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.589964    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 13:07:52.590841 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.590830    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 13:07:52.594936 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.594924    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 13:07:52.595065 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.595056    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 13:07:52.595096 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.595088    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 13:07:52.595137 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.595103    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 13:07:52.595137 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.595121    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 13:07:52.595137 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.595133    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 03 13:07:52.595288 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.595145    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 13:07:52.595618 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.595599    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 13:07:52.598367 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.598327    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 13:07:52.603709 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.603692    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 13:07:52.604486 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.604466    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 13:07:52.604567 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.604499    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 13:07:52.604648 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.604637    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 13:07:52.604682 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.604653    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 13:07:52.604682 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.604659    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 13:07:52.604682 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.604665    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 13:07:52.604682 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.604671    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 13:07:52.604682 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.604677    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 13:07:52.605143 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.605126    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 13:07:52.605143 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.605135    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 13:07:52.605143 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.605142    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 13:07:52.605226 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.605148    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 13:07:52.605226 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.605157    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 13:07:52.605226 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.605167    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 13:07:52.606176 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.606164    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 13:07:52.606221 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.606177    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 13:07:52.609569 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.609548    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rpf4w"
Nov 03 13:07:52.610308 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.610297    2577 server.go:1263] "Started kubelet"
Nov 03 13:07:52.610398 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.610359    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 13:07:52.610455 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.610399    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 13:07:52.610512 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.610495    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 13:07:52.611258 ip-10-0-132-167 systemd[1]: Started Kubernetes Kubelet.
Nov 03 13:07:52.611506 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.611313    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-167.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 13:07:52.611917 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.611666    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 13:07:52.611917 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.611805    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-167.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 13:07:52.613865 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.613836    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 13:07:52.614535 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.614521    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 13:07:52.616004 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.614975    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-167.us-east-2.compute.internal.18e48ae6218c8a01  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-167.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-167.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 13:07:52.610269697 +0000 UTC m=+0.353136163,LastTimestamp:2026-11-03 13:07:52.610269697 +0000 UTC m=+0.353136163,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-167.us-east-2.compute.internal,}"
Nov 03 13:07:52.617136 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.617118    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rpf4w"
Nov 03 13:07:52.617962 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.617939    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 13:07:52.618574 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.618553    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 13:07:52.619619 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.619603    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 13:07:52.619735 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.619717    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:52.620230 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620103    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 13:07:52.620230 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620121    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 13:07:52.620230 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620109    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 13:07:52.620230 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620211    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 13:07:52.620230 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620220    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 13:07:52.620513 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620451    2577 factory.go:55] Registering systemd factory
Nov 03 13:07:52.620513 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620464    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 03 13:07:52.620644 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620633    2577 factory.go:153] Registering CRI-O factory
Nov 03 13:07:52.620644 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620643    2577 factory.go:223] Registration of the crio container factory successfully
Nov 03 13:07:52.620738 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620687    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 13:07:52.620738 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620701    2577 factory.go:103] Registering Raw factory
Nov 03 13:07:52.620738 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.620710    2577 manager.go:1199] Started watching for new ooms in manager
Nov 03 13:07:52.621631 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.621614    2577 manager.go:319] Starting recovery of all containers
Nov 03 13:07:52.632992 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.632974    2577 manager.go:324] Recovery completed
Nov 03 13:07:52.635579 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.635557    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:07:52.637711 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.637697    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:07:52.638876 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.638858    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-167.us-east-2.compute.internal\" not found" node="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.639992 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.639964    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:07:52.639992 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.639994    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:07:52.640270 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.640004    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:07:52.640619 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.640604    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 03 13:07:52.640619 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.640616    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 13:07:52.640735 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.640633    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 13:07:52.640735 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.640635    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:07:52.641320 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.641304    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:07:52.641409 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.641332    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:07:52.641409 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.641358    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:07:52.642602 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.642590    2577 policy_none.go:50] "Start"
Nov 03 13:07:52.642643 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.642611    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 13:07:52.642643 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.642621    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 13:07:52.644527 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.644510    2577 policy_none.go:44] "Start"
Nov 03 13:07:52.667913 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.667889    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 13:07:52.670213 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.670196    2577 manager.go:341] "Starting Device Plugin manager"
Nov 03 13:07:52.670304 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.670222    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 13:07:52.670304 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.670230    2577 server.go:87] "Starting device plugin registration server"
Nov 03 13:07:52.670498 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.670481    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 13:07:52.670574 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.670533    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 13:07:52.670769 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.670753    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 13:07:52.670881 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.670836    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 13:07:52.670881 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.670845    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 13:07:52.671245 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.671226    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 13:07:52.671387 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.671275    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:52.766791 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.766760    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 13:07:52.767964 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.767948    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 13:07:52.768026 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.767970    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 13:07:52.768026 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.767996    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 13:07:52.768123 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.768036    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 13:07:52.770706 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.770684    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:07:52.771517 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.771496    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:07:52.771612 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.771535    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:07:52.771612 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.771550    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:07:52.771612 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.771585    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.772571 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.772556    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:07:52.780380 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.780364    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.780462 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.780390    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-167.us-east-2.compute.internal\": node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:52.806205 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.806159    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:52.868595 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.868537    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"]
Nov 03 13:07:52.868692 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.868631    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:07:52.869594 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.869579    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:07:52.869662 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.869609    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:07:52.869662 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.869620    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:07:52.871806 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.871793    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:07:52.871940 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.871927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.871975 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.871957    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:07:52.872534 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.872515    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:07:52.872596 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.872544    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:07:52.872596 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.872553    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:07:52.872596 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.872518    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:07:52.872687 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.872615    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:07:52.872687 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.872628    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:07:52.874717 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.874699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.874810 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.874728    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:07:52.875373 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.875356    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:07:52.875458 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.875385    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:07:52.875458 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.875398    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:07:52.892729 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.892714    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found" node="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.894683 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.894668    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found" node="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.907109 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:52.907093    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:52.922079 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.922062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.922130 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.922085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:52.922130 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:52.922103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c4c57a57b1c5fc6c67e60988666a273-config\") pod \"kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"6c4c57a57b1c5fc6c67e60988666a273\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.007927 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:53.007767    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:53.022353 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.022321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.022411 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.022361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c4c57a57b1c5fc6c67e60988666a273-config\") pod \"kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"6c4c57a57b1c5fc6c67e60988666a273\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.022411 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.022378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.022473 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.022424    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.022473 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.022428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.022473 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.022428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c4c57a57b1c5fc6c67e60988666a273-config\") pod \"kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"6c4c57a57b1c5fc6c67e60988666a273\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.108825 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:53.108753    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:53.196322 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.196286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.198431 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.198410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.209494 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:53.209469    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:53.309972 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:53.309941    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:53.410530 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:53.410474    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:53.511999 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:53.511958    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:53.555472 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.555442    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 13:07:53.555900 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.555595    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 13:07:53.555900 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.555599    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 13:07:53.612117 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:53.612084    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:53.620646 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.620592    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 13:02:53 +0000 UTC" deadline="2028-05-09 01:03:38.443180606 +0000 UTC"
Nov 03 13:07:53.620646 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.620643    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13259h55m44.822540865s"
Nov 03 13:07:53.713200 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:53.713170    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:53.813394 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:53.813362    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-167.us-east-2.compute.internal\" not found"
Nov 03 13:07:53.816389 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:53.816354    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba9b1d6fba3e1fb24d7e6543c829290.slice/crio-b17899e6d9fb087950134182a897dac8c7a3c3d3c7d29562cdfb2c9bdb57f62f WatchSource:0}: Error finding container b17899e6d9fb087950134182a897dac8c7a3c3d3c7d29562cdfb2c9bdb57f62f: Status 404 returned error can't find the container with id b17899e6d9fb087950134182a897dac8c7a3c3d3c7d29562cdfb2c9bdb57f62f
Nov 03 13:07:53.823378 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.823361    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:07:53.828577 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:53.828556    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4c57a57b1c5fc6c67e60988666a273.slice/crio-a224f8518796cc0c15709c56f6578d9c820474c1527c6ec4d12439b0b6ccb1ec WatchSource:0}: Error finding container a224f8518796cc0c15709c56f6578d9c820474c1527c6ec4d12439b0b6ccb1ec: Status 404 returned error can't find the container with id a224f8518796cc0c15709c56f6578d9c820474c1527c6ec4d12439b0b6ccb1ec
Nov 03 13:07:53.839473 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.839456    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:07:53.920730 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.920695    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.933228 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.933208    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:07:53.935386 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.935372    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:07:53.944858 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:53.944816    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:07:54.077040 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.077007    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:07:54.596391 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.595995    2577 apiserver.go:52] "Watching apiserver"
Nov 03 13:07:54.605284 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.605256    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 13:07:54.606442 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.606383    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal","openshift-multus/multus-mkzmk","kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-btbj2","openshift-multus/multus-additional-cni-plugins-gjtf7","openshift-network-operator/iptables-alerter-2fwt9","openshift-dns/node-resolver-fgqkh","openshift-image-registry/node-ca-88kk5","openshift-multus/network-metrics-daemon-2m9tv","openshift-network-diagnostics/network-check-target-ch6h8","openshift-ovn-kubernetes/ovnkube-node-62dpn","kube-system/global-pull-secret-syncer-9wrnl","kube-system/konnectivity-agent-d9722","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"]
Nov 03 13:07:54.609095 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.609068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.611598 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.611576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:07:54.613747 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.613723    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 13:07:54.613856 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.613848    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-knm89\""
Nov 03 13:07:54.613931 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.613912    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 13:07:54.615408 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.614007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.615408 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.615171    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 13:07:54.615408 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.615283    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vwt8x\""
Nov 03 13:07:54.615408 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.615385    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 13:07:54.616276 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.616256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.617633 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.617611    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 13:07:54.617714 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.617683    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 13:07:54.617781 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.617744    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 13:07:54.617838 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.617817    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jqlk5\""
Nov 03 13:07:54.618968 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.618945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.619439 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.619407    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 13:07:54.619690 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.619670    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 13:07:54.619801 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.619775    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:07:54.619912 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.619804    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s4cl2\""
Nov 03 13:07:54.621289 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.621270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.622299 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.622281    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 13:07:54.622535 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.622516    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wv2nv\""
Nov 03 13:07:54.622535 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.622523    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 13:07:54.622887 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.622769    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 13:07:54.622887 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.622791    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 13:07:54.622887 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.622860    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 13:07:54.624239 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.624218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:54.624357 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.624330    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:07:54.626371 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.626285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.627131 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.627111    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 13:07:54.628898 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.628878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.631535 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.631514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:07:54.631632 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.631581    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:07:54.632943 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.632918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.633048 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.632961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw6jc\" (UniqueName: \"kubernetes.io/projected/69042048-b2e6-477d-a330-94d2a1e608e3-kube-api-access-nw6jc\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.633048 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.632991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-kubernetes\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.633048 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-tuned\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.633215 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-hosts-file\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.633215 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-conf\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.633215 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-system-cni-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.633215 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.633215 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.633544 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.633544 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-lib-modules\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.633544 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-tmp\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.633544 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.633544 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2e2e850-03eb-485d-940e-1b6d9063091c-iptables-alerter-script\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.633544 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aaa0065-2805-432d-8c2e-ef03591b6c90-agent-certs\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:07:54.633544 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa0065-2805-432d-8c2e-ef03591b6c90-konnectivity-ca\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:07:54.633544 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.634002 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-modprobe-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.634002 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-var-lib-kubelet\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.634002 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.634002 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.633745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-registration-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.634209 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634070    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.634500 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysconfig\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.634595 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-sys\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.634643 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-host\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.634683 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-cnibin\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.634771 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-sys-fs\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.634816 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmwl\" (UniqueName: \"kubernetes.io/projected/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-kube-api-access-lrmwl\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.634862 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-systemd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.634917 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-run\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.634917 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6bhd\" (UniqueName: \"kubernetes.io/projected/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-kube-api-access-n6bhd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.635023 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-tmp-dir\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.635023 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.634996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg2kd\" (UniqueName: \"kubernetes.io/projected/4da03b13-3a30-48de-b399-ba82b071c456-kube-api-access-mg2kd\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.635109 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.635035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2e2e850-03eb-485d-940e-1b6d9063091c-host-slash\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.635109 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.635062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-socket-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.635193 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.635109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-device-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.635193 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.635136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czgz\" (UniqueName: \"kubernetes.io/projected/b2e2e850-03eb-485d-940e-1b6d9063091c-kube-api-access-4czgz\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.636791 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.636767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:54.636892 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.636841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:07:54.662624 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.662601    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 13:07:54.662712 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.662671    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m2v66"
Nov 03 13:07:54.672708 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.672685    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m2v66"
Nov 03 13:07:54.721797 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.721771    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 13:07:54.722379 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.722357    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 13:07:54.722618 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.722592    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 13:07:54.722724 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.722675    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c596b\""
Nov 03 13:07:54.722897 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.722879    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-62cxl\""
Nov 03 13:07:54.723001 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.722976    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 13:07:54.723061 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.723007    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 13:07:54.723105 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.723089    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 13:07:54.723178 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.723165    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 13:07:54.723233 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.723193    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 13:07:54.723313 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.723296    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 13:07:54.723432 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.723327    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 13:07:54.723880 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.723854    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 13:07:54.724008 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.723859    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:07:54.724167 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.724143    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-psqgf\""
Nov 03 13:07:54.724247 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.724172    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mrd6\""
Nov 03 13:07:54.725358 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.725323    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 13:07:54.735651 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-kubelet\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.735765 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-dbus\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:54.735765 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:54.735765 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-socket-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.735874 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:54.735874 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-bin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.735874 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-multus\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.735874 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t69vq\" (UniqueName: \"kubernetes.io/projected/378ce7f3-c33f-44f5-a383-a8acc714f547-kube-api-access-t69vq\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.735874 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.736087 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw6jc\" (UniqueName: \"kubernetes.io/projected/69042048-b2e6-477d-a330-94d2a1e608e3-kube-api-access-nw6jc\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.736087 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735902    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-socket-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.736087 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.735920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-system-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.736087 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736026    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.736260 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-netns\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.736297 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-systemd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.736297 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-node-log\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.736467 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.736467 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba1ed75b-f38b-4589-87fa-0c8d22abe784-host\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.736467 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-kubelet\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.736467 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.736657 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-lib-modules\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.736657 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-tmp\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.736657 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.736657 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2e2e850-03eb-485d-940e-1b6d9063091c-iptables-alerter-script\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.736657 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9k4v\" (UniqueName: \"kubernetes.io/projected/ba1ed75b-f38b-4589-87fa-0c8d22abe784-kube-api-access-l9k4v\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.736891 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqb6m\" (UniqueName: \"kubernetes.io/projected/d237eda3-f867-4dd3-8a9b-24aaa9c37403-kube-api-access-tqb6m\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:54.736891 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cni-binary-copy\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.736891 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aaa0065-2805-432d-8c2e-ef03591b6c90-agent-certs\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:07:54.736891 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa0065-2805-432d-8c2e-ef03591b6c90-konnectivity-ca\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:07:54.736891 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.736891 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-modprobe-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-var-lib-kubelet\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-etc-kubernetes\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.736986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-netns\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-sys\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg67\" (UniqueName: \"kubernetes.io/projected/b6257887-6b6e-4ad3-ae76-3d121659e2d4-kube-api-access-shg67\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-run\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-k8s-cni-cncf-io\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-log-socket\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-bin\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-script-lib\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-conf-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737163    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.737192 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-device-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737270    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czgz\" (UniqueName: \"kubernetes.io/projected/b2e2e850-03eb-485d-940e-1b6d9063091c-kube-api-access-4czgz\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-kubernetes\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-tuned\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-daemon-config\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-hosts-file\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-conf\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-system-cni-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-hostroot\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737787    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-device-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-multus-certs\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-slash\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.737946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-socket-dir-parent\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737890    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-var-lib-kubelet\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-systemd-units\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-env-overrides\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737921    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-sys\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737858    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-modprobe-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.737982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/378ce7f3-c33f-44f5-a383-a8acc714f547-ovn-node-metrics-cert\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cnibin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-netd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-kubelet-config\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-registration-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysconfig\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-host\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-cnibin\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-var-lib-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.739123 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-config\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-sys-fs\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmwl\" (UniqueName: \"kubernetes.io/projected/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-kube-api-access-lrmwl\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-systemd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bhd\" (UniqueName: \"kubernetes.io/projected/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-kube-api-access-n6bhd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba1ed75b-f38b-4589-87fa-0c8d22abe784-serviceca\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-etc-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-ovn\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-tmp-dir\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg2kd\" (UniqueName: \"kubernetes.io/projected/4da03b13-3a30-48de-b399-ba82b071c456-kube-api-access-mg2kd\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2e2e850-03eb-485d-940e-1b6d9063091c-host-slash\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738437    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-hosts-file\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-cnibin\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738531    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-host\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738554    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysconfig\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738570    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-registration-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738604    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738651    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.739815 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738746    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.738796    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-run\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739095    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-tmp-dir\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739247    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2e2e850-03eb-485d-940e-1b6d9063091c-iptables-alerter-script\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739299    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-kubernetes\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739333    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-systemd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739393    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa0065-2805-432d-8c2e-ef03591b6c90-konnectivity-ca\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739396    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-conf\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739414    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739483    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-system-cni-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739484    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-sys-fs\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739515    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2e2e850-03eb-485d-940e-1b6d9063091c-host-slash\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.740405 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.739528    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-lib-modules\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.742569 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.742548    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-tuned\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.742780 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.742745    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aaa0065-2805-432d-8c2e-ef03591b6c90-agent-certs\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:07:54.744739 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.744717    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-tmp\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.745197 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.745061    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw6jc\" (UniqueName: \"kubernetes.io/projected/69042048-b2e6-477d-a330-94d2a1e608e3-kube-api-access-nw6jc\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.747402 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.747356    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg2kd\" (UniqueName: \"kubernetes.io/projected/4da03b13-3a30-48de-b399-ba82b071c456-kube-api-access-mg2kd\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.747646 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.747621    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czgz\" (UniqueName: \"kubernetes.io/projected/b2e2e850-03eb-485d-940e-1b6d9063091c-kube-api-access-4czgz\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.748128 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.748035    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmwl\" (UniqueName: \"kubernetes.io/projected/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-kube-api-access-lrmwl\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.749564 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.749543    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bhd\" (UniqueName: \"kubernetes.io/projected/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-kube-api-access-n6bhd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.773865 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.773820    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"6c4c57a57b1c5fc6c67e60988666a273","Type":"ContainerStarted","Data":"a224f8518796cc0c15709c56f6578d9c820474c1527c6ec4d12439b0b6ccb1ec"}
Nov 03 13:07:54.775270 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.775245    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"bba9b1d6fba3e1fb24d7e6543c829290","Type":"ContainerStarted","Data":"b17899e6d9fb087950134182a897dac8c7a3c3d3c7d29562cdfb2c9bdb57f62f"}
Nov 03 13:07:54.795704 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.795671    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:07:54.839515 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-kubelet\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.839694 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-dbus\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:54.839694 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:54.839694 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:54.839694 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839640    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-kubelet\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.839694 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839666    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-dbus\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:54.839694 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.839684    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-bin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-multus\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.839770    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:55.339746943 +0000 UTC m=+3.082613422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839776    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-multus\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.839781    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t69vq\" (UniqueName: \"kubernetes.io/projected/378ce7f3-c33f-44f5-a383-a8acc714f547-kube-api-access-t69vq\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839818    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-bin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-system-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.839844    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret podName:0ef0c534-57ad-4269-8d12-e2a5879a3eed nodeName:}" failed. No retries permitted until 2026-11-03 13:07:55.339826576 +0000 UTC m=+3.082693047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret") pod "global-pull-secret-syncer-9wrnl" (UID: "0ef0c534-57ad-4269-8d12-e2a5879a3eed") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-netns\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-systemd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-node-log\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba1ed75b-f38b-4589-87fa-0c8d22abe784-host\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-kubelet\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9k4v\" (UniqueName: \"kubernetes.io/projected/ba1ed75b-f38b-4589-87fa-0c8d22abe784-kube-api-access-l9k4v\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.840043 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqb6m\" (UniqueName: \"kubernetes.io/projected/d237eda3-f867-4dd3-8a9b-24aaa9c37403-kube-api-access-tqb6m\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840070    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-netns\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.839883    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-system-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cni-binary-copy\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-etc-kubernetes\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840133    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba1ed75b-f38b-4589-87fa-0c8d22abe784-host\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840180    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-node-log\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840220    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-kubelet\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840259    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-etc-kubernetes\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840264    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-systemd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-netns\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840487    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-netns\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg67\" (UniqueName: \"kubernetes.io/projected/b6257887-6b6e-4ad3-ae76-3d121659e2d4-kube-api-access-shg67\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-k8s-cni-cncf-io\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840691    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cni-binary-copy\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-log-socket\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840716    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-k8s-cni-cncf-io\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.840857 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-bin\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-log-socket\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-script-lib\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840788    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-bin\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-conf-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-daemon-config\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840945    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-conf-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-hostroot\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-multus-certs\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-slash\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841031    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841076    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-multus-certs\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-socket-dir-parent\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-systemd-units\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.841776 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841144    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-script-lib\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-env-overrides\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/378ce7f3-c33f-44f5-a383-a8acc714f547-ovn-node-metrics-cert\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-hostroot\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cnibin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-netd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841286    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-systemd-units\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-kubelet-config\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-var-lib-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-config\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba1ed75b-f38b-4589-87fa-0c8d22abe784-serviceca\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841456    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-netd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841229    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-slash\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841267    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-socket-dir-parent\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841497    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841504    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cnibin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841514    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842462 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.840995    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-etc-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841551    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-kubelet-config\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841576    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-etc-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-ovn\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841609    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-var-lib-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841663    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-env-overrides\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841675    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-daemon-config\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841710    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-ovn\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.841895    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba1ed75b-f38b-4589-87fa-0c8d22abe784-serviceca\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.842980 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.842023    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-config\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.843876 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.843841    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/378ce7f3-c33f-44f5-a383-a8acc714f547-ovn-node-metrics-cert\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.851050 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.850984    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:07:54.851050 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.851027    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:07:54.851050 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.851043    2577 projected.go:196] Error preparing data for projected volume kube-api-access-w9rrw for pod openshift-network-diagnostics/network-check-target-ch6h8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:07:54.851281 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:54.851113    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw podName:70f3bdac-e826-4187-b83c-993fe2c5d7ac nodeName:}" failed. No retries permitted until 2026-11-03 13:07:55.351091412 +0000 UTC m=+3.093957891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w9rrw" (UniqueName: "kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw") pod "network-check-target-ch6h8" (UID: "70f3bdac-e826-4187-b83c-993fe2c5d7ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:07:54.851281 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.851274    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t69vq\" (UniqueName: \"kubernetes.io/projected/378ce7f3-c33f-44f5-a383-a8acc714f547-kube-api-access-t69vq\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:54.853145 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.853122    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg67\" (UniqueName: \"kubernetes.io/projected/b6257887-6b6e-4ad3-ae76-3d121659e2d4-kube-api-access-shg67\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.853459 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.853425    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9k4v\" (UniqueName: \"kubernetes.io/projected/ba1ed75b-f38b-4589-87fa-0c8d22abe784-kube-api-access-l9k4v\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.853551 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.853449    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqb6m\" (UniqueName: \"kubernetes.io/projected/d237eda3-f867-4dd3-8a9b-24aaa9c37403-kube-api-access-tqb6m\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:54.925327 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.925285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:07:54.935851 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.935824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:07:54.941765 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:54.941732    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f5fcad_a6fb_4bd2_99aa_f4973415aafa.slice/crio-ad0cdcc8dcdd9f4f7a79ab7de95b4f095710ca0b7e17481f2173ec2db307ee03 WatchSource:0}: Error finding container ad0cdcc8dcdd9f4f7a79ab7de95b4f095710ca0b7e17481f2173ec2db307ee03: Status 404 returned error can't find the container with id ad0cdcc8dcdd9f4f7a79ab7de95b4f095710ca0b7e17481f2173ec2db307ee03
Nov 03 13:07:54.946715 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.946694    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:07:54.947862 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:54.947840    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aaa0065_2805_432d_8c2e_ef03591b6c90.slice/crio-4e596cffd821982e01aac8bf393a629ff7f669f1e46d898bbbfe943af8ea7574 WatchSource:0}: Error finding container 4e596cffd821982e01aac8bf393a629ff7f669f1e46d898bbbfe943af8ea7574: Status 404 returned error can't find the container with id 4e596cffd821982e01aac8bf393a629ff7f669f1e46d898bbbfe943af8ea7574
Nov 03 13:07:54.949470 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.949448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:07:54.954535 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.954512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:07:54.961385 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:54.961353    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69042048_b2e6_477d_a330_94d2a1e608e3.slice/crio-0d31311f07b072c3954f643b71cfeeb311b29eb278ea4cb97ac487b085ee457b WatchSource:0}: Error finding container 0d31311f07b072c3954f643b71cfeeb311b29eb278ea4cb97ac487b085ee457b: Status 404 returned error can't find the container with id 0d31311f07b072c3954f643b71cfeeb311b29eb278ea4cb97ac487b085ee457b
Nov 03 13:07:54.967409 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:54.966706    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586f11e1_e0bd_439c_9aa3_2cb5eb3d2e12.slice/crio-a84d8d978fe5227d2836472e71d766a4ac5fc9b6ca345127777cee0e3a2c0563 WatchSource:0}: Error finding container a84d8d978fe5227d2836472e71d766a4ac5fc9b6ca345127777cee0e3a2c0563: Status 404 returned error can't find the container with id a84d8d978fe5227d2836472e71d766a4ac5fc9b6ca345127777cee0e3a2c0563
Nov 03 13:07:54.974234 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.974215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:07:54.976627 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.976604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:07:54.978737 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.978715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:07:54.987530 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:54.987503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mkzmk"
Nov 03 13:07:54.991309 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:54.991279    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da03b13_3a30_48de_b399_ba82b071c456.slice/crio-5f36c5f908a0e3ecfd15dfb0c1b32d37e42eaaccb99fe1e7e505befb895958d7 WatchSource:0}: Error finding container 5f36c5f908a0e3ecfd15dfb0c1b32d37e42eaaccb99fe1e7e505befb895958d7: Status 404 returned error can't find the container with id 5f36c5f908a0e3ecfd15dfb0c1b32d37e42eaaccb99fe1e7e505befb895958d7
Nov 03 13:07:54.992622 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:54.992581    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e2e850_03eb_485d_940e_1b6d9063091c.slice/crio-1d1571b49cb1c7df6bbc7557d742185cf9872c169a868e5c43f82e3e914809af WatchSource:0}: Error finding container 1d1571b49cb1c7df6bbc7557d742185cf9872c169a868e5c43f82e3e914809af: Status 404 returned error can't find the container with id 1d1571b49cb1c7df6bbc7557d742185cf9872c169a868e5c43f82e3e914809af
Nov 03 13:07:54.996766 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:54.996740    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1ed75b_f38b_4589_87fa_0c8d22abe784.slice/crio-b08bac6b69c078563f5cd4962940ec2ddd902be2e677e3ca4cfc212fe2dfbd04 WatchSource:0}: Error finding container b08bac6b69c078563f5cd4962940ec2ddd902be2e677e3ca4cfc212fe2dfbd04: Status 404 returned error can't find the container with id b08bac6b69c078563f5cd4962940ec2ddd902be2e677e3ca4cfc212fe2dfbd04
Nov 03 13:07:55.000196 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:55.000134    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6257887_6b6e_4ad3_ae76_3d121659e2d4.slice/crio-74156cec001d800de4637071ad0f1fa24e2707746dcd54ebeac0c88a9b40a298 WatchSource:0}: Error finding container 74156cec001d800de4637071ad0f1fa24e2707746dcd54ebeac0c88a9b40a298: Status 404 returned error can't find the container with id 74156cec001d800de4637071ad0f1fa24e2707746dcd54ebeac0c88a9b40a298
Nov 03 13:07:55.008436 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.008407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:07:55.020331 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:07:55.020284    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378ce7f3_c33f_44f5_a383_a8acc714f547.slice/crio-e993aaafb8602fa337c3e113c05717357b281013437acf7c97292ba70a4e011d WatchSource:0}: Error finding container e993aaafb8602fa337c3e113c05717357b281013437acf7c97292ba70a4e011d: Status 404 returned error can't find the container with id e993aaafb8602fa337c3e113c05717357b281013437acf7c97292ba70a4e011d
Nov 03 13:07:55.344006 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.343969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:55.344145 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.344027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:55.344210 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.344171    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:07:55.344265 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.344238    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:56.344218281 +0000 UTC m=+4.087084754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:07:55.344647 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.344625    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:07:55.344755 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.344687    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret podName:0ef0c534-57ad-4269-8d12-e2a5879a3eed nodeName:}" failed. No retries permitted until 2026-11-03 13:07:56.344670177 +0000 UTC m=+4.087536650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret") pod "global-pull-secret-syncer-9wrnl" (UID: "0ef0c534-57ad-4269-8d12-e2a5879a3eed") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:07:55.444990 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.444929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:07:55.445159 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.445060    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:07:55.445159 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.445082    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:07:55.445159 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.445097    2577 projected.go:196] Error preparing data for projected volume kube-api-access-w9rrw for pod openshift-network-diagnostics/network-check-target-ch6h8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:07:55.445314 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.445179    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw podName:70f3bdac-e826-4187-b83c-993fe2c5d7ac nodeName:}" failed. No retries permitted until 2026-11-03 13:07:56.445158748 +0000 UTC m=+4.188025227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w9rrw" (UniqueName: "kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw") pod "network-check-target-ch6h8" (UID: "70f3bdac-e826-4187-b83c-993fe2c5d7ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:07:55.676845 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.676526    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 13:02:55 +0000 UTC" deadline="2028-05-03 13:38:19.140771419 +0000 UTC"
Nov 03 13:07:55.677696 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.676852    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13128h30m23.46392797s"
Nov 03 13:07:55.768953 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.768919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:55.769142 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.769059    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:07:55.769540 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.769516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:55.769669 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.769647    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:07:55.769746 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.769732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:07:55.769831 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:55.769814    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:07:55.793234 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.792342    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"6c4c57a57b1c5fc6c67e60988666a273","Type":"ContainerStarted","Data":"ecb0265ba4916a75056aa397d5216c66ea3435be0ff769038c8380f541439858"}
Nov 03 13:07:55.805440 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.804810    2577 generic.go:356] "Generic (PLEG): container finished" podID="bba9b1d6fba3e1fb24d7e6543c829290" containerID="a49a9057c0dddeda639211f58c4e5ff2f85c73164fd32c20816c40acf6ecc1db" exitCode=0
Nov 03 13:07:55.805440 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.804894    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"bba9b1d6fba3e1fb24d7e6543c829290","Type":"ContainerDied","Data":"a49a9057c0dddeda639211f58c4e5ff2f85c73164fd32c20816c40acf6ecc1db"}
Nov 03 13:07:55.811545 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.810933    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal" podStartSLOduration=1.810916835 podStartE2EDuration="1.810916835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:55.80956254 +0000 UTC m=+3.552429020" watchObservedRunningTime="2026-11-03 13:07:55.810916835 +0000 UTC m=+3.553783315"
Nov 03 13:07:55.811545 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.811274    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" event={"ID":"378ce7f3-c33f-44f5-a383-a8acc714f547","Type":"ContainerStarted","Data":"e993aaafb8602fa337c3e113c05717357b281013437acf7c97292ba70a4e011d"}
Nov 03 13:07:55.823912 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.823777    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-88kk5" event={"ID":"ba1ed75b-f38b-4589-87fa-0c8d22abe784","Type":"ContainerStarted","Data":"b08bac6b69c078563f5cd4962940ec2ddd902be2e677e3ca4cfc212fe2dfbd04"}
Nov 03 13:07:55.826666 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.826529    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2fwt9" event={"ID":"b2e2e850-03eb-485d-940e-1b6d9063091c","Type":"ContainerStarted","Data":"1d1571b49cb1c7df6bbc7557d742185cf9872c169a868e5c43f82e3e914809af"}
Nov 03 13:07:55.840502 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.840437    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-btbj2" event={"ID":"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12","Type":"ContainerStarted","Data":"a84d8d978fe5227d2836472e71d766a4ac5fc9b6ca345127777cee0e3a2c0563"}
Nov 03 13:07:55.845337 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.845309    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9722" event={"ID":"2aaa0065-2805-432d-8c2e-ef03591b6c90","Type":"ContainerStarted","Data":"4e596cffd821982e01aac8bf393a629ff7f669f1e46d898bbbfe943af8ea7574"}
Nov 03 13:07:55.847367 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.847343    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fgqkh" event={"ID":"44f5fcad-a6fb-4bd2-99aa-f4973415aafa","Type":"ContainerStarted","Data":"ad0cdcc8dcdd9f4f7a79ab7de95b4f095710ca0b7e17481f2173ec2db307ee03"}
Nov 03 13:07:55.854184 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.854161    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mkzmk" event={"ID":"b6257887-6b6e-4ad3-ae76-3d121659e2d4","Type":"ContainerStarted","Data":"74156cec001d800de4637071ad0f1fa24e2707746dcd54ebeac0c88a9b40a298"}
Nov 03 13:07:55.856181 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.856154    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjtf7" event={"ID":"4da03b13-3a30-48de-b399-ba82b071c456","Type":"ContainerStarted","Data":"5f36c5f908a0e3ecfd15dfb0c1b32d37e42eaaccb99fe1e7e505befb895958d7"}
Nov 03 13:07:55.871581 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:55.871554    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf" event={"ID":"69042048-b2e6-477d-a330-94d2a1e608e3","Type":"ContainerStarted","Data":"0d31311f07b072c3954f643b71cfeeb311b29eb278ea4cb97ac487b085ee457b"}
Nov 03 13:07:56.353370 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:56.353285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:56.353370 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:56.353336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:56.353628 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:56.353514    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:07:56.353628 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:56.353582    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:58.353561198 +0000 UTC m=+6.096427671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:07:56.354012 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:56.353990    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:07:56.354097 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:56.354053    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret podName:0ef0c534-57ad-4269-8d12-e2a5879a3eed nodeName:}" failed. No retries permitted until 2026-11-03 13:07:58.354036473 +0000 UTC m=+6.096902942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret") pod "global-pull-secret-syncer-9wrnl" (UID: "0ef0c534-57ad-4269-8d12-e2a5879a3eed") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:07:56.454544 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:56.454504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:07:56.454745 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:56.454698    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:07:56.454745 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:56.454723    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:07:56.454745 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:56.454736    2577 projected.go:196] Error preparing data for projected volume kube-api-access-w9rrw for pod openshift-network-diagnostics/network-check-target-ch6h8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:07:56.454963 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:56.454795    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw podName:70f3bdac-e826-4187-b83c-993fe2c5d7ac nodeName:}" failed. No retries permitted until 2026-11-03 13:07:58.454777505 +0000 UTC m=+6.197643976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w9rrw" (UniqueName: "kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw") pod "network-check-target-ch6h8" (UID: "70f3bdac-e826-4187-b83c-993fe2c5d7ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:07:56.678143 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:56.678016    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 13:02:55 +0000 UTC" deadline="2028-06-10 20:20:53.240416703 +0000 UTC"
Nov 03 13:07:56.678143 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:56.678062    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14047h12m56.562360593s"
Nov 03 13:07:56.925800 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:56.925763    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"bba9b1d6fba3e1fb24d7e6543c829290","Type":"ContainerStarted","Data":"f5b44ff0d716cdce2a0136f69d33cdb8d1f5e686057d184b0dd1b484bbc84d2c"}
Nov 03 13:07:56.943979 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:56.943880    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal" podStartSLOduration=2.943863188 podStartE2EDuration="2.943863188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:56.943381738 +0000 UTC m=+4.686248218" watchObservedRunningTime="2026-11-03 13:07:56.943863188 +0000 UTC m=+4.686729667"
Nov 03 13:07:57.768802 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:57.768749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:57.769312 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:57.768892    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:07:57.769382 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:57.769321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:57.769510 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:57.769484    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:07:57.769592 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:57.769560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:07:57.769653 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:57.769634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:07:58.371948 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:58.371905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:58.372144 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:58.371970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:58.372223 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:58.372197    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:07:58.372282 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:58.372276    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:02.372253023 +0000 UTC m=+10.115119503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:07:58.372790 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:58.372757    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:07:58.372886 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:58.372819    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret podName:0ef0c534-57ad-4269-8d12-e2a5879a3eed nodeName:}" failed. No retries permitted until 2026-11-03 13:08:02.372801916 +0000 UTC m=+10.115668409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret") pod "global-pull-secret-syncer-9wrnl" (UID: "0ef0c534-57ad-4269-8d12-e2a5879a3eed") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:07:58.472416 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:58.472309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:07:58.472575 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:58.472543    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:07:58.472575 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:58.472565    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:07:58.472686 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:58.472578    2577 projected.go:196] Error preparing data for projected volume kube-api-access-w9rrw for pod openshift-network-diagnostics/network-check-target-ch6h8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:07:58.472686 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:58.472641    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw podName:70f3bdac-e826-4187-b83c-993fe2c5d7ac nodeName:}" failed. No retries permitted until 2026-11-03 13:08:02.472622751 +0000 UTC m=+10.215489211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w9rrw" (UniqueName: "kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw") pod "network-check-target-ch6h8" (UID: "70f3bdac-e826-4187-b83c-993fe2c5d7ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:07:58.945393 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:58.945343    2577 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="1d2f5a33df48981cf2aa09979a1c1c02307eb0be606b113d2b5a567f600c7c4a" exitCode=0
Nov 03 13:07:58.945880 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:58.945433    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjtf7" event={"ID":"4da03b13-3a30-48de-b399-ba82b071c456","Type":"ContainerDied","Data":"1d2f5a33df48981cf2aa09979a1c1c02307eb0be606b113d2b5a567f600c7c4a"}
Nov 03 13:07:59.768778 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:59.768745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:07:59.768935 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:59.768861    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:07:59.769205 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:59.769187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:07:59.769285 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:59.769271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:07:59.769346 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:59.769307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:07:59.769414 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:07:59.769360    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:07:59.954177 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:07:59.953397    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9722" event={"ID":"2aaa0065-2805-432d-8c2e-ef03591b6c90","Type":"ContainerStarted","Data":"ec8434863cacea8e6eb128a69f4a68e509a524eb0f71b6566397b6e4d0e1007c"}
Nov 03 13:08:00.964480 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:00.963151    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-88kk5" event={"ID":"ba1ed75b-f38b-4589-87fa-0c8d22abe784","Type":"ContainerStarted","Data":"4bf5a7c72c0720f2c819571f8a4f0448256c1ade24db651bb5d2181c167ffd67"}
Nov 03 13:08:00.979553 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:00.979479    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d9722" podStartSLOduration=4.057137516 podStartE2EDuration="7.979447774s" totalImagesPullingTime="3.922310258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:59.972409989 +0000 UTC m=+7.715276469" watchObservedRunningTime="2026-11-03 13:08:00.979447774 +0000 UTC m=+8.722314250"
Nov 03 13:08:01.769137 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:01.769096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:01.769334 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:01.769254    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:01.769744 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:01.769723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:01.769869 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:01.769844    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:01.769921 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:01.769913    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:01.770007 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:01.769986    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:01.969304 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:01.969207    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf" event={"ID":"69042048-b2e6-477d-a330-94d2a1e608e3","Type":"ContainerStarted","Data":"8e4783e0e34e4a4d53bc957951791fa8c3796324072a46247f5968abf77ddf18"}
Nov 03 13:08:02.406721 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:02.406681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:02.406721 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:02.406726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:02.406959 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:02.406886    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:02.406959 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:02.406945    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:10.406932137 +0000 UTC m=+18.149798605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:02.407359 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:02.407341    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:02.407475 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:02.407390    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret podName:0ef0c534-57ad-4269-8d12-e2a5879a3eed nodeName:}" failed. No retries permitted until 2026-11-03 13:08:10.407376201 +0000 UTC m=+18.150242673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret") pod "global-pull-secret-syncer-9wrnl" (UID: "0ef0c534-57ad-4269-8d12-e2a5879a3eed") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:02.507588 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:02.507543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:02.507799 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:02.507780    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:08:02.507857 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:02.507813    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:08:02.507857 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:02.507829    2577 projected.go:196] Error preparing data for projected volume kube-api-access-w9rrw for pod openshift-network-diagnostics/network-check-target-ch6h8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:02.507937 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:02.507897    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw podName:70f3bdac-e826-4187-b83c-993fe2c5d7ac nodeName:}" failed. No retries permitted until 2026-11-03 13:08:10.50787775 +0000 UTC m=+18.250744228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w9rrw" (UniqueName: "kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw") pod "network-check-target-ch6h8" (UID: "70f3bdac-e826-4187-b83c-993fe2c5d7ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:02.801362 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:02.801311    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 13:08:02.974626 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:02.974538    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf" event={"ID":"69042048-b2e6-477d-a330-94d2a1e608e3","Type":"ContainerStarted","Data":"cbba8cf3cafedf844a7bb00fef08b2c6c86594b9b700b304cb5aced4b68a00a4"}
Nov 03 13:08:03.678026 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.677287    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:08:02.801329166Z","UUID":"d9dd81fe-66c4-436f-8c16-cec0b288a838","Handler":null,"Name":"","Endpoint":""}
Nov 03 13:08:03.679674 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.679317    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 13:08:03.679674 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.679366    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 13:08:03.768388 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.768309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:03.768515 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:03.768471    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:03.768897 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.768879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:03.769007 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:03.768985    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:03.769079 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.769065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:03.769164 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:03.769148    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:03.979747 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.979290    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fgqkh" event={"ID":"44f5fcad-a6fb-4bd2-99aa-f4973415aafa","Type":"ContainerStarted","Data":"df4995fca1d8847cd235e935eb8f901eecc0d8b9c214228cc2a8a7d8589ffa6d"}
Nov 03 13:08:03.981235 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.981210    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mkzmk" event={"ID":"b6257887-6b6e-4ad3-ae76-3d121659e2d4","Type":"ContainerStarted","Data":"f51a59741216ee9d16026bfae7d82baa0c6d681113064142b3ed99403c02b0e1"}
Nov 03 13:08:03.983816 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.983789    2577 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="853a68e1545e888f2bb55c609f8d8a52b1570b8ccf656f1198b34ff9b750f38b" exitCode=0
Nov 03 13:08:03.983922 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.983831    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjtf7" event={"ID":"4da03b13-3a30-48de-b399-ba82b071c456","Type":"ContainerDied","Data":"853a68e1545e888f2bb55c609f8d8a52b1570b8ccf656f1198b34ff9b750f38b"}
Nov 03 13:08:03.994906 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.994864    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-fgqkh" podStartSLOduration=2.376243057 podStartE2EDuration="10.994847524s" totalImagesPullingTime="8.618604467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:03.994692214 +0000 UTC m=+11.737558688" watchObservedRunningTime="2026-11-03 13:08:03.994847524 +0000 UTC m=+11.737713999"
Nov 03 13:08:03.995047 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:03.994947    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-88kk5" podStartSLOduration=5.543656619 podStartE2EDuration="10.994940343s" totalImagesPullingTime="5.451283724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:00.97977544 +0000 UTC m=+8.722641921" watchObservedRunningTime="2026-11-03 13:08:03.994940343 +0000 UTC m=+11.737806845"
Nov 03 13:08:04.015324 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:04.015262    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mkzmk" podStartSLOduration=2.968168539 podStartE2EDuration="11.01524548s" totalImagesPullingTime="8.047076941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:04.01451013 +0000 UTC m=+11.757376608" watchObservedRunningTime="2026-11-03 13:08:04.01524548 +0000 UTC m=+11.758111956"
Nov 03 13:08:04.119912 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:04.119838    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:08:04.120553 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:04.120368    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:08:04.990058 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:04.990003    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf" event={"ID":"69042048-b2e6-477d-a330-94d2a1e608e3","Type":"ContainerStarted","Data":"1d50525efe77f2635756fa033552f36ab52ae11104e51f20ca88ab728f39af80"}
Nov 03 13:08:04.991842 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:04.991813    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2fwt9" event={"ID":"b2e2e850-03eb-485d-940e-1b6d9063091c","Type":"ContainerStarted","Data":"53eb2e492419b6fe01a92e3cea48abfc00f72a931e2e3b9834a7a51204fdcf9e"}
Nov 03 13:08:05.011063 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:05.010964    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf" podStartSLOduration=3.3707368 podStartE2EDuration="12.010937463s" totalImagesPullingTime="8.640200663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:05.008237077 +0000 UTC m=+12.751103550" watchObservedRunningTime="2026-11-03 13:08:05.010937463 +0000 UTC m=+12.753803916"
Nov 03 13:08:05.024580 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:05.024522    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2fwt9" podStartSLOduration=3.716645898 podStartE2EDuration="12.024502718s" totalImagesPullingTime="8.30785682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:05.024304875 +0000 UTC m=+12.767171352" watchObservedRunningTime="2026-11-03 13:08:05.024502718 +0000 UTC m=+12.767369194"
Nov 03 13:08:05.769199 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:05.769114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:05.769199 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:05.769164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:05.769419 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:05.769266    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:05.769419 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:05.769315    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:05.769419 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:05.769334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:05.769534 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:05.769442    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:05.995361 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:05.995309    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-btbj2" event={"ID":"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12","Type":"ContainerStarted","Data":"7625e4543f6918b92640fd56b7d461aa242aa2f300542cd8e8f6b879332bfbab"}
Nov 03 13:08:05.998338 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:05.998309    2577 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="7b4a691a7a3c68c24da9250e660e31e21e384beef09d6bf716c49970056ca5c7" exitCode=0
Nov 03 13:08:05.998470 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:05.998385    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjtf7" event={"ID":"4da03b13-3a30-48de-b399-ba82b071c456","Type":"ContainerDied","Data":"7b4a691a7a3c68c24da9250e660e31e21e384beef09d6bf716c49970056ca5c7"}
Nov 03 13:08:06.012746 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:06.012691    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-btbj2" podStartSLOduration=2.479826927 podStartE2EDuration="13.012672716s" totalImagesPullingTime="10.532845789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:06.012197134 +0000 UTC m=+13.755063609" watchObservedRunningTime="2026-11-03 13:08:06.012672716 +0000 UTC m=+13.755539192"
Nov 03 13:08:07.768788 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:07.768574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:07.769260 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:07.768583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:07.769260 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:07.768870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:07.769260 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:07.768607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:07.769260 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:07.768972    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:07.769260 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:07.769025    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:08.006239 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:08.006200    2577 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="d24ffd5a9e79619a78a0b70edb7a3a73ff9dc6c2b1e419c93e17104267cb69c3" exitCode=0
Nov 03 13:08:08.006422 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:08.006266    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjtf7" event={"ID":"4da03b13-3a30-48de-b399-ba82b071c456","Type":"ContainerDied","Data":"d24ffd5a9e79619a78a0b70edb7a3a73ff9dc6c2b1e419c93e17104267cb69c3"}
Nov 03 13:08:09.768542 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:09.768513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:09.768542 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:09.768540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:09.768978 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:09.768613    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:09.768978 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:09.768671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:09.768978 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:09.768736    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:09.768978 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:09.768797    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:10.013851 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:10.013530    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" event={"ID":"378ce7f3-c33f-44f5-a383-a8acc714f547","Type":"ContainerStarted","Data":"9ccaf6d5aaaf4a426e6af93f0b0229443a69e8f8bd9a6c68ebd680d5d877b042"}
Nov 03 13:08:10.013851 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:10.013793    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" event={"ID":"378ce7f3-c33f-44f5-a383-a8acc714f547","Type":"ContainerStarted","Data":"b47a1cf4eac83d3c9718319da95e674cefdb1cc5bc062be9465d0670fd6b8da8"}
Nov 03 13:08:10.465621 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:10.465588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:10.465621 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:10.465628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:10.465849 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:10.465750    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:10.465849 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:10.465771    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:10.465849 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:10.465811    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:26.465797788 +0000 UTC m=+34.208664256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:10.465849 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:10.465831    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret podName:0ef0c534-57ad-4269-8d12-e2a5879a3eed nodeName:}" failed. No retries permitted until 2026-11-03 13:08:26.465818614 +0000 UTC m=+34.208685081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret") pod "global-pull-secret-syncer-9wrnl" (UID: "0ef0c534-57ad-4269-8d12-e2a5879a3eed") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:10.566551 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:10.566515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:10.566727 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:10.566707    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:08:10.566787 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:10.566730    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:08:10.566787 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:10.566743    2577 projected.go:196] Error preparing data for projected volume kube-api-access-w9rrw for pod openshift-network-diagnostics/network-check-target-ch6h8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:10.566879 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:10.566809    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw podName:70f3bdac-e826-4187-b83c-993fe2c5d7ac nodeName:}" failed. No retries permitted until 2026-11-03 13:08:26.566789945 +0000 UTC m=+34.309656415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w9rrw" (UniqueName: "kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw") pod "network-check-target-ch6h8" (UID: "70f3bdac-e826-4187-b83c-993fe2c5d7ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:11.017434 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:11.017338    2577 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="30bc73e0c269e19f3f427c70243e2ed6cc5ffac962f566b9607baf6ea3e7004f" exitCode=0
Nov 03 13:08:11.017891 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:11.017433    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjtf7" event={"ID":"4da03b13-3a30-48de-b399-ba82b071c456","Type":"ContainerDied","Data":"30bc73e0c269e19f3f427c70243e2ed6cc5ffac962f566b9607baf6ea3e7004f"}
Nov 03 13:08:11.020376 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:11.020336    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" event={"ID":"378ce7f3-c33f-44f5-a383-a8acc714f547","Type":"ContainerStarted","Data":"bfe293a4ac7fefb64f6379f032cdff212aa49c89b1e359b0295b3d40195363ff"}
Nov 03 13:08:11.020457 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:11.020443    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" event={"ID":"378ce7f3-c33f-44f5-a383-a8acc714f547","Type":"ContainerStarted","Data":"01b1b76b194d707dc930e65cef23c7542bd6fbda62fc2e80d17d51850a73a3fe"}
Nov 03 13:08:11.020524 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:11.020480    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" event={"ID":"378ce7f3-c33f-44f5-a383-a8acc714f547","Type":"ContainerStarted","Data":"580a8b24710abe0b9c1447772c2e639a663b62bad763c3c08b870fd7e00415cf"}
Nov 03 13:08:11.020524 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:11.020510    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" event={"ID":"378ce7f3-c33f-44f5-a383-a8acc714f547","Type":"ContainerStarted","Data":"89390c12e83caee83532bbedb21cfe3711015c6144f85ad422859da7159618d3"}
Nov 03 13:08:11.768685 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:11.768656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:11.768685 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:11.768682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:11.768885 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:11.768753    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:11.768885 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:11.768794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:11.768885 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:11.768872    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:11.768974 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:11.768943    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:12.025168 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:12.025080    2577 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="a13cdc30eadcd5e2b06d6053b692401cf49aec9d07c8a9cb2489c4fde2858fb9" exitCode=0
Nov 03 13:08:12.025168 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:12.025113    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjtf7" event={"ID":"4da03b13-3a30-48de-b399-ba82b071c456","Type":"ContainerDied","Data":"a13cdc30eadcd5e2b06d6053b692401cf49aec9d07c8a9cb2489c4fde2858fb9"}
Nov 03 13:08:13.029879 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:13.029841    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" event={"ID":"378ce7f3-c33f-44f5-a383-a8acc714f547","Type":"ContainerStarted","Data":"54f6941f8e6da4ef8ef8c6b37a4047c974590f9a10b315a4300229b103a5d1cf"}
Nov 03 13:08:13.032791 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:13.032767    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjtf7" event={"ID":"4da03b13-3a30-48de-b399-ba82b071c456","Type":"ContainerStarted","Data":"9e96f0b4b1ea42659dddc3d33ab031e0a1d54c94c0248667dd00cdd79347ec33"}
Nov 03 13:08:13.055167 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:13.055127    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gjtf7" podStartSLOduration=7.770357549 podStartE2EDuration="20.055115499s" totalImagesPullingTime="12.28475795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:13.053442156 +0000 UTC m=+20.796308631" watchObservedRunningTime="2026-11-03 13:08:13.055115499 +0000 UTC m=+20.797981974"
Nov 03 13:08:13.768876 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:13.768846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:13.769058 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:13.768948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:13.769058 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:13.768958    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:13.769058 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:13.768976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:13.769058 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:13.769052    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:13.769273 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:13.769117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:15.040037 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.039841    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" event={"ID":"378ce7f3-c33f-44f5-a383-a8acc714f547","Type":"ContainerStarted","Data":"9ba44710d92d41b50227105c677999a42682dad91b58964a269de079a89ec93d"}
Nov 03 13:08:15.040674 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.040130    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:15.040674 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.040186    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:15.040674 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.040235    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:15.055659 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.055632    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:15.055783 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.055756    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:15.070732 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.070690    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn" podStartSLOduration=7.222859956 podStartE2EDuration="22.070677879s" totalImagesPullingTime="14.847817923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:15.068772273 +0000 UTC m=+22.811638747" watchObservedRunningTime="2026-11-03 13:08:15.070677879 +0000 UTC m=+22.813544354"
Nov 03 13:08:15.769109 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.769077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:15.769109 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.769102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:15.769305 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:15.769179    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:15.769305 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:15.769227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:15.769305 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:15.769290    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:15.769436 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:15.769401    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:16.176614 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:16.176540    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9wrnl"]
Nov 03 13:08:16.177235 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:16.177216    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ch6h8"]
Nov 03 13:08:16.177440 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:16.177425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:16.177594 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:16.177288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:16.177709 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:16.177635    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:16.177709 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:16.177695    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:16.177946 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:16.177927    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2m9tv"]
Nov 03 13:08:16.178014 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:16.178001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:16.178106 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:16.178087    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:17.768803 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:17.768767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:17.769262 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:17.768866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:17.769262 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:17.768869    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:17.769262 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:17.768874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:17.769262 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:17.768963    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:17.769262 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:17.769004    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:19.768759 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:19.768723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:19.768759 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:19.768748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:19.769328 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:19.768870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:19.769328 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:19.768902    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:19.769328 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:19.768940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:19.769328 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:19.769012    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:21.769010 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:21.768839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:21.769430 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:21.768896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:21.769430 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:21.769084    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wrnl" podUID="0ef0c534-57ad-4269-8d12-e2a5879a3eed"
Nov 03 13:08:21.769430 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:21.769163    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch6h8" podUID="70f3bdac-e826-4187-b83c-993fe2c5d7ac"
Nov 03 13:08:21.769430 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:21.768896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:21.769430 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:21.769247    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2m9tv" podUID="d237eda3-f867-4dd3-8a9b-24aaa9c37403"
Nov 03 13:08:22.182021 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.181993    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:08:22.182217 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.182204    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 13:08:22.182618 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.182600    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:08:22.583519 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.583491    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeReady"
Nov 03 13:08:22.583675 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.583666    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 13:08:22.624146 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.624116    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x7hff"]
Nov 03 13:08:22.628423 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.628388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:22.632011 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.631984    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v5xss"]
Nov 03 13:08:22.632548 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.632525    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 13:08:22.632697 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.632671    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 13:08:22.632697 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.632697    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 13:08:22.632855 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.632789    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gkggt\""
Nov 03 13:08:22.636699 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.636176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.636790 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.636768    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7hff"]
Nov 03 13:08:22.645527 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.645502    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v5xss"]
Nov 03 13:08:22.736122 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.736049    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 13:08:22.736122 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.736082    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 13:08:22.736122 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.736094    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wd8kt\""
Nov 03 13:08:22.753384 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.753332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4100403-c8f8-4ac1-98c4-2795d5a11b10-tmp-dir\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.753557 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.753462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4100403-c8f8-4ac1-98c4-2795d5a11b10-config-volume\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.753557 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.753492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrblj\" (UniqueName: \"kubernetes.io/projected/b4100403-c8f8-4ac1-98c4-2795d5a11b10-kube-api-access-vrblj\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.753557 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.753510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.753557 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.753531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-cert\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:22.753703 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.753624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llfvj\" (UniqueName: \"kubernetes.io/projected/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-kube-api-access-llfvj\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:22.854458 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.854416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4100403-c8f8-4ac1-98c4-2795d5a11b10-tmp-dir\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.854920 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.854471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4100403-c8f8-4ac1-98c4-2795d5a11b10-config-volume\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.854920 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.854493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrblj\" (UniqueName: \"kubernetes.io/projected/b4100403-c8f8-4ac1-98c4-2795d5a11b10-kube-api-access-vrblj\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.854920 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.854655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.854920 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.854712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-cert\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:22.854920 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.854759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfvj\" (UniqueName: \"kubernetes.io/projected/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-kube-api-access-llfvj\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:22.854920 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:22.854764    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 13:08:22.854920 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:22.854864    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls podName:b4100403-c8f8-4ac1-98c4-2795d5a11b10 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:23.354844061 +0000 UTC m=+31.097710527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls") pod "dns-default-v5xss" (UID: "b4100403-c8f8-4ac1-98c4-2795d5a11b10") : secret "dns-default-metrics-tls" not found
Nov 03 13:08:22.855213 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.855111    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4100403-c8f8-4ac1-98c4-2795d5a11b10-tmp-dir\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.855309 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.855288    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4100403-c8f8-4ac1-98c4-2795d5a11b10-config-volume\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.858660 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.858639    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-cert\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:22.863802 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.863778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrblj\" (UniqueName: \"kubernetes.io/projected/b4100403-c8f8-4ac1-98c4-2795d5a11b10-kube-api-access-vrblj\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:22.864153 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.864136    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfvj\" (UniqueName: \"kubernetes.io/projected/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-kube-api-access-llfvj\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:22.943665 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:22.943632    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:23.228430 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.228396    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7hff"]
Nov 03 13:08:23.231586 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:08:23.231556    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bc4a29_7ff7_4b22_8f9f_56906b70ee41.slice/crio-d92b628d953dcfd79bbb5c3ca497b658f08a7e262d5ac583c4e8f6ed6e9c1d8c WatchSource:0}: Error finding container d92b628d953dcfd79bbb5c3ca497b658f08a7e262d5ac583c4e8f6ed6e9c1d8c: Status 404 returned error can't find the container with id d92b628d953dcfd79bbb5c3ca497b658f08a7e262d5ac583c4e8f6ed6e9c1d8c
Nov 03 13:08:23.358169 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.358078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:23.358309 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:23.358235    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 13:08:23.358309 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:23.358300    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls podName:b4100403-c8f8-4ac1-98c4-2795d5a11b10 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:24.358282746 +0000 UTC m=+32.101149217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls") pod "dns-default-v5xss" (UID: "b4100403-c8f8-4ac1-98c4-2795d5a11b10") : secret "dns-default-metrics-tls" not found
Nov 03 13:08:23.769279 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.769240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:23.769279 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.769272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:23.769521 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.769466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:23.774039 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.774019    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 13:08:23.774155 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.774057    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 13:08:23.774155 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.774095    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 13:08:23.774155 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.774149    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d95tp\""
Nov 03 13:08:23.774299 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.774152    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-569wf\""
Nov 03 13:08:23.774299 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.774177    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hltl5\""
Nov 03 13:08:23.774299 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:23.774210    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 13:08:24.060645 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:24.060567    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7hff" event={"ID":"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41","Type":"ContainerStarted","Data":"d92b628d953dcfd79bbb5c3ca497b658f08a7e262d5ac583c4e8f6ed6e9c1d8c"}
Nov 03 13:08:24.365396 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:24.365307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:24.365556 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:24.365476    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 13:08:24.365556 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:24.365551    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls podName:b4100403-c8f8-4ac1-98c4-2795d5a11b10 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:26.365530916 +0000 UTC m=+34.108397375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls") pod "dns-default-v5xss" (UID: "b4100403-c8f8-4ac1-98c4-2795d5a11b10") : secret "dns-default-metrics-tls" not found
Nov 03 13:08:25.292146 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.292110    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dqvjq"]
Nov 03 13:08:25.312010 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.311982    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dqvjq"]
Nov 03 13:08:25.312167 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.312029    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-f9xld-h6jgs"]
Nov 03 13:08:25.312167 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.312137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.327561 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.327539    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f9xld-h6jgs"]
Nov 03 13:08:25.327696 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.327644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.375092 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.375057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.375255 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.375121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58899f4-efb6-4d21-b1eb-fcea8c046356-data-volume\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.375255 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.375179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjw8\" (UniqueName: \"kubernetes.io/projected/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-api-access-cpjw8\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.375336 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.375291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d58899f4-efb6-4d21-b1eb-fcea8c046356-crio-socket\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.375336 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.375313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.405984 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.405956    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 13:08:25.406156 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.406073    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 13:08:25.406156 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.406139    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 13:08:25.406285 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.406199    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 13:08:25.406285 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.406240    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s2gms\""
Nov 03 13:08:25.406285 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.406256    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-sjjkt\""
Nov 03 13:08:25.406427 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.406297    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 13:08:25.411688 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.411665    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 13:08:25.475802 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.475760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.475984 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.475815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58899f4-efb6-4d21-b1eb-fcea8c046356-data-volume\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.475984 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.475834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjw8\" (UniqueName: \"kubernetes.io/projected/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-api-access-cpjw8\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.475984 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.475862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.475984 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.475908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d58899f4-efb6-4d21-b1eb-fcea8c046356-crio-socket\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.475984 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.475934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.475984 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:25.475941    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:25.475984 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.475975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xwvb\" (UniqueName: \"kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.476333 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.476003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.476333 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:25.476019    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:25.975997446 +0000 UTC m=+33.718863918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:25.476333 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.476062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.476333 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.476155    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58899f4-efb6-4d21-b1eb-fcea8c046356-data-volume\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.476333 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:25.476163    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 13:08:25.476333 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:25.476215    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:25.976200596 +0000 UTC m=+33.719067068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : configmap "kube-rbac-proxy" not found
Nov 03 13:08:25.476333 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.476269    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d58899f4-efb6-4d21-b1eb-fcea8c046356-crio-socket\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.484568 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.484537    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjw8\" (UniqueName: \"kubernetes.io/projected/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-api-access-cpjw8\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.577282 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.577189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.577282 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.577268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xwvb\" (UniqueName: \"kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.577576 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.577294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.577576 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.577324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.577699 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.577681    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.577843 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.577819    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.578630 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.578606    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.585775 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.585752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xwvb\" (UniqueName: \"kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.676049 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.676008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:25.980651 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.980616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.980816 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:25.980698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:25.980816 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:25.980771    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 13:08:25.980888 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:25.980846    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:26.980830249 +0000 UTC m=+34.723696732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : configmap "kube-rbac-proxy" not found
Nov 03 13:08:25.980888 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:25.980853    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:25.980976 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:25.980914    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:26.980901018 +0000 UTC m=+34.723767493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:26.028438 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.028408    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f9xld-h6jgs"]
Nov 03 13:08:26.074692 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:08:26.074661    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb93242_cc6c_4a41_bd2b_fa73aca1af1f.slice/crio-5e7c3e44d06308063cee8fa279e17a7654386af7e2096777d3cff63364e7901b WatchSource:0}: Error finding container 5e7c3e44d06308063cee8fa279e17a7654386af7e2096777d3cff63364e7901b: Status 404 returned error can't find the container with id 5e7c3e44d06308063cee8fa279e17a7654386af7e2096777d3cff63364e7901b
Nov 03 13:08:26.383642 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.383599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:26.384007 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:26.383650    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 13:08:26.384007 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:26.383712    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls podName:b4100403-c8f8-4ac1-98c4-2795d5a11b10 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:30.383698839 +0000 UTC m=+38.126565309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls") pod "dns-default-v5xss" (UID: "b4100403-c8f8-4ac1-98c4-2795d5a11b10") : secret "dns-default-metrics-tls" not found
Nov 03 13:08:26.484312 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.484227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:26.484312 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.484267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:26.484559 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:26.484452    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 13:08:26.484559 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:26.484517    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:58.484503286 +0000 UTC m=+66.227369739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : secret "metrics-daemon-secret" not found
Nov 03 13:08:26.486622 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.486599    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:26.516103 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.516068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:26.584862 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.584823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:26.587579 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.587554    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:26.792772 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.792681    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9wrnl"]
Nov 03 13:08:26.793540 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.793518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:26.796324 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:08:26.796294    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef0c534_57ad_4269_8d12_e2a5879a3eed.slice/crio-de6a626e38afd4e614b20f6bbb5dd5955d5a422ac21f07ef66539a93e7857232 WatchSource:0}: Error finding container de6a626e38afd4e614b20f6bbb5dd5955d5a422ac21f07ef66539a93e7857232: Status 404 returned error can't find the container with id de6a626e38afd4e614b20f6bbb5dd5955d5a422ac21f07ef66539a93e7857232
Nov 03 13:08:26.987649 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.987608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:26.987839 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.987699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:26.987839 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:26.987779    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:26.987945 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:26.987848    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:28.987830581 +0000 UTC m=+36.730697050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:26.988278 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:26.988250    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:27.068838 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:27.068751    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f9xld-h6jgs" event={"ID":"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f","Type":"ContainerStarted","Data":"5e7c3e44d06308063cee8fa279e17a7654386af7e2096777d3cff63364e7901b"}
Nov 03 13:08:27.070714 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:27.070631    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7hff" event={"ID":"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41","Type":"ContainerStarted","Data":"b6faae6b5b86b3b8aad2f1f6d5e4e54848f72f386e91b48aa43c10056e5b0518"}
Nov 03 13:08:27.071917 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:27.071853    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9wrnl" event={"ID":"0ef0c534-57ad-4269-8d12-e2a5879a3eed","Type":"ContainerStarted","Data":"de6a626e38afd4e614b20f6bbb5dd5955d5a422ac21f07ef66539a93e7857232"}
Nov 03 13:08:27.086779 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:27.086724    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x7hff" podStartSLOduration=2.212889678 podStartE2EDuration="5.086707265s" totalImagesPullingTime="2.873817587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:27.086155753 +0000 UTC m=+34.829022229" watchObservedRunningTime="2026-11-03 13:08:27.086707265 +0000 UTC m=+34.829573744"
Nov 03 13:08:27.090912 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:27.090889    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ch6h8"]
Nov 03 13:08:27.094621 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:08:27.094592    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f3bdac_e826_4187_b83c_993fe2c5d7ac.slice/crio-6d99166880f3ba57c3dd45e928f430c8e4c88dc84c2e7713c96f8f3b4e9208eb WatchSource:0}: Error finding container 6d99166880f3ba57c3dd45e928f430c8e4c88dc84c2e7713c96f8f3b4e9208eb: Status 404 returned error can't find the container with id 6d99166880f3ba57c3dd45e928f430c8e4c88dc84c2e7713c96f8f3b4e9208eb
Nov 03 13:08:28.077561 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:28.077528    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ch6h8" event={"ID":"70f3bdac-e826-4187-b83c-993fe2c5d7ac","Type":"ContainerStarted","Data":"6d99166880f3ba57c3dd45e928f430c8e4c88dc84c2e7713c96f8f3b4e9208eb"}
Nov 03 13:08:29.006603 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:29.006516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:29.006745 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:29.006658    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:29.006789 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:29.006744    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:33.006721146 +0000 UTC m=+40.749587615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:29.082386 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:29.082324    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f9xld-h6jgs" event={"ID":"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f","Type":"ContainerStarted","Data":"ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c"}
Nov 03 13:08:29.100767 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:29.100711    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-f9xld-h6jgs" podStartSLOduration=1.455025784 podStartE2EDuration="4.100692702s" totalImagesPullingTime="2.645666918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:29.099985825 +0000 UTC m=+36.842852301" watchObservedRunningTime="2026-11-03 13:08:29.100692702 +0000 UTC m=+36.843559178"
Nov 03 13:08:30.385454 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:30.385426    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fgqkh_44f5fcad-a6fb-4bd2-99aa-f4973415aafa/dns-node-resolver/0.log"
Nov 03 13:08:30.419781 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:30.419741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:30.419958 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:30.419881    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 13:08:30.420023 ip-10-0-132-167 kubenswrapper[2577]: E1103 13:08:30.419964    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls podName:b4100403-c8f8-4ac1-98c4-2795d5a11b10 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:38.419944581 +0000 UTC m=+46.162811048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls") pod "dns-default-v5xss" (UID: "b4100403-c8f8-4ac1-98c4-2795d5a11b10") : secret "dns-default-metrics-tls" not found
Nov 03 13:08:31.090915 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.090826    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ch6h8" event={"ID":"70f3bdac-e826-4187-b83c-993fe2c5d7ac","Type":"ContainerStarted","Data":"1beb15ddba410f103244a7a831edcfe343bd32796d7e6cedeb4b1474e79f153b"}
Nov 03 13:08:31.091057 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.090966    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:31.108458 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.108395    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ch6h8" podStartSLOduration=34.500842483 podStartE2EDuration="38.108376251s" totalImagesPullingTime="3.607533768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:31.107610607 +0000 UTC m=+38.850477083" watchObservedRunningTime="2026-11-03 13:08:31.108376251 +0000 UTC m=+38.851242728"
Nov 03 13:08:31.382279 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.382203    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"]
Nov 03 13:08:31.410097 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.410061    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"]
Nov 03 13:08:31.410541 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.410197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"
Nov 03 13:08:31.510148 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.510112    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 13:08:31.510325 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.510207    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zrpxj\""
Nov 03 13:08:31.510325 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.510261    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 13:08:31.528719 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.528687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vflt4\" (UniqueName: \"kubernetes.io/projected/1b6197ff-1085-4ade-a095-688ff39c5f9c-kube-api-access-vflt4\") pod \"migrator-66df5d496b-6b6r5\" (UID: \"1b6197ff-1085-4ade-a095-688ff39c5f9c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"
Nov 03 13:08:31.585592 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.585567    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-88kk5_ba1ed75b-f38b-4589-87fa-0c8d22abe784/node-ca/0.log"
Nov 03 13:08:31.630058 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.630020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vflt4\" (UniqueName: \"kubernetes.io/projected/1b6197ff-1085-4ade-a095-688ff39c5f9c-kube-api-access-vflt4\") pod \"migrator-66df5d496b-6b6r5\" (UID: \"1b6197ff-1085-4ade-a095-688ff39c5f9c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"
Nov 03 13:08:31.638410 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.638332    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vflt4\" (UniqueName: \"kubernetes.io/projected/1b6197ff-1085-4ade-a095-688ff39c5f9c-kube-api-access-vflt4\") pod \"migrator-66df5d496b-6b6r5\" (UID: \"1b6197ff-1085-4ade-a095-688ff39c5f9c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"
Nov 03 13:08:31.742825 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:31.742782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"
Nov 03 13:08:32.087803 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:32.087763    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"]
Nov 03 13:08:32.090930 ip-10-0-132-167 kubenswrapper[2577]: W1103 13:08:32.090902    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6197ff_1085_4ade_a095_688ff39c5f9c.slice/crio-a7e20460e2b520467b392129ef9ef9d2cc4f478329e0a431d39313a958ac7224 WatchSource:0}: Error finding container a7e20460e2b520467b392129ef9ef9d2cc4f478329e0a431d39313a958ac7224: Status 404 returned error can't find the container with id a7e20460e2b520467b392129ef9ef9d2cc4f478329e0a431d39313a958ac7224
Nov 03 13:08:32.096026 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:32.095997    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9wrnl" event={"ID":"0ef0c534-57ad-4269-8d12-e2a5879a3eed","Type":"ContainerStarted","Data":"e92d591cddc0d23a72bcec868502b1210be3ea7873fa6bf7d18ca9edd893be87"}
Nov 03 13:08:32.115385 ip-10-0-132-167 kubenswrapper[2577]: I1103 13:08:32.115314    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9wrnl" podStartSLOduration=34.08440413 podStartE2EDuration="39.115293776s" totalImagesPullingTime="5.030889646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:32.114738465 +0000 UTC m=+39.857604939" watchObservedRunningTime="2026-11-03 13:08:32.115293776 +0000 UTC m=+39.858160252"
Nov 03 13:08:32.131276 ip-10-0-132-167 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 13:08:32.141777 ip-10-0-132-167 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 13:08:32.141944 ip-10-0-132-167 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 13:08:32.141992 ip-10-0-132-167 systemd[1]: kubelet.service: Consumed 2.295s CPU time.
Nov 03 13:08:32.157993 ip-10-0-132-167 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 13:08:32.227187 ip-10-0-132-167 kubenswrapper[4447]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:08:32.227187 ip-10-0-132-167 kubenswrapper[4447]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 13:08:32.227187 ip-10-0-132-167 kubenswrapper[4447]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:08:32.227187 ip-10-0-132-167 kubenswrapper[4447]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:08:32.229298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229217    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:32.229298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229297    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:32.229298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229301    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229304    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229307    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229310    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229313    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229315    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229318    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229322    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229325    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229328    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229331    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229333    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229336    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229339    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229358    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229361    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229363    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229366    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229368    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229371    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:32.229407 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229374    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229376    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229379    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229382    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229385    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229392    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229395    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229398    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229401    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229403    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229406    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229409    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229412    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229415    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229418    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229421    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229426    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229430    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229434    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:32.229896 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229438    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229441    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229444    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229446    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229449    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229451    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229454    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229457    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229459    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229462    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229465    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229467    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229470    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229472    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229475    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229478    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229481    4447 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229483    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229486    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:32.230384 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229490    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229492    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229495    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229498    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229500    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229504    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229507    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229510    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229512    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229514    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229517    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229520    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229523    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229525    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229528    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229530    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229532    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229535    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229537    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229540    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:32.230845 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229543    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229545    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229548    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229550    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229553    4447 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229555    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229558    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229561    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229565    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229569    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229573    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229576    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229578    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229581    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229584    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229587    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229589    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229592    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229594    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:32.231324 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229597    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229600    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229602    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229604    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229608    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229610    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.229613    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230356    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230363    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230366    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230369    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230372    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230375    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230377    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230381    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230383    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230386    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230389    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230392    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:32.231812 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230395    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230398    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230401    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230404    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230406    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230409    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230411    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230414    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230418    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230421    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230424    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230426    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230429    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230431    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230434    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230436    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230439    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230441    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230444    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230446    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:32.232322 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230449    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230451    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230454    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230456    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230459    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230461    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230465    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230468    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230470    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230473    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230475    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230478    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230480    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230483    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230486    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230488    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230491    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230493    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230496    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230499    4447 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:32.232827 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230502    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230507    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230509    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230514    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230519    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230522    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230525    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230528    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230531    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230533    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230536    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230539    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230542    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230544    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230547    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230549    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230552    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230555    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230558    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:32.233314 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230560    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230563    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230565    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230568    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230571    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230573    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230576    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230578    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230581    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230584    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230586    4447 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230589    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230591    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230594    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230596    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230600    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230602    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230605    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230607    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230610    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:32.233830 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230612    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230615    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230617    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230620    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230622    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230624    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230627    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230630    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230633    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230635    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230638    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230640    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230643    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230645    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.230649    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231114    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231122    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231127    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231131    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:32.234316 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231135    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231139    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231144    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231148    4447 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231151    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231154    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231157    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231159    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231164    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231167    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231169    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231172    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231174    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231177    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231180    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231182    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231185    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231189    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231193    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231195    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:32.234926 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231198    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231201    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231203    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231206    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231209    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231211    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231214    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231216    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231219    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231222    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231224    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231227    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231229    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231232    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231234    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231237    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231239    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231242    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231244    4447 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231247    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:32.235423 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231249    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231252    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231255    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231257    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231260    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231263    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231266    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231268    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231273    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231276    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231279    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231281    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231284    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231286    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231289    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231292    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231296    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231300    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:32.235903 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231303    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231306    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231309    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231311    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231314    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231316    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231319    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231321    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231324    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231326    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231329    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231332    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231335    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231337    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231352    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231355    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231358    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231361    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231364    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231366    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:32.236437 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231369    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231372    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231375    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231378    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231380    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231383    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231385    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231388    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231391    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231393    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231396    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231398    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231401    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231403    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231406    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231409    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231412    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231415    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231417    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231420    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:32.237283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231422    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231426    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231429    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.231431    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231521    4447 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231532    4447 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231538    4447 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231543    4447 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231547    4447 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231551    4447 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231556    4447 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231560    4447 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231563    4447 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231566    4447 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231571    4447 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231575    4447 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231578    4447 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231581    4447 flags.go:64] FLAG: --cgroup-root=""
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231584    4447 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231588    4447 flags.go:64] FLAG: --client-ca-file=""
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231590    4447 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231594    4447 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231597    4447 flags.go:64] FLAG: --cluster-domain=""
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231601    4447 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 13:08:32.237837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231604    4447 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231607    4447 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231611    4447 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231615    4447 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231618    4447 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231621    4447 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231625    4447 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231628    4447 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231631    4447 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231634    4447 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231637    4447 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231640    4447 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231644    4447 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231647    4447 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231650    4447 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231653    4447 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231656    4447 flags.go:64] FLAG: --enable-server="true"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231659    4447 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231664    4447 flags.go:64] FLAG: --event-burst="100"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231667    4447 flags.go:64] FLAG: --event-qps="50"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231670    4447 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231673    4447 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231677    4447 flags.go:64] FLAG: --eviction-hard=""
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231681    4447 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 13:08:32.238464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231685    4447 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231688    4447 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231691    4447 flags.go:64] FLAG: --eviction-soft=""
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231694    4447 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231697    4447 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231700    4447 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231704    4447 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231707    4447 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231710    4447 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231713    4447 flags.go:64] FLAG: --feature-gates=""
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231717    4447 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231720    4447 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231723    4447 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231726    4447 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231729    4447 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231732    4447 flags.go:64] FLAG: --help="false"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231735    4447 flags.go:64] FLAG: --hostname-override="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231739    4447 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231742    4447 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231745    4447 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231749    4447 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231752    4447 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231755    4447 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 13:08:32.239154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231758    4447 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231761    4447 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231764    4447 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231767    4447 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231770    4447 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231773    4447 flags.go:64] FLAG: --kube-reserved=""
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231776    4447 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231781    4447 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231784    4447 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231786    4447 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231789    4447 flags.go:64] FLAG: --lock-file=""
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231792    4447 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231795    4447 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231798    4447 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231804    4447 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231807    4447 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231811    4447 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231813    4447 flags.go:64] FLAG: --logging-format="text"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231817    4447 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231820    4447 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231823    4447 flags.go:64] FLAG: --manifest-url=""
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231826    4447 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231831    4447 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231834    4447 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231838    4447 flags.go:64] FLAG: --max-pods="110"
Nov 03 13:08:32.239735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231841    4447 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231844    4447 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231847    4447 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231849    4447 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231852    4447 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231855    4447 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231858    4447 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231865    4447 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231868    4447 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231871    4447 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231874    4447 flags.go:64] FLAG: --pod-cidr=""
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231877    4447 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231887    4447 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231891    4447 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231894    4447 flags.go:64] FLAG: --port="10250"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231897    4447 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231900    4447 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-062da26ab9667c414"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231903    4447 flags.go:64] FLAG: --qos-reserved=""
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231906    4447 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231910    4447 flags.go:64] FLAG: --register-node="true"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231913    4447 flags.go:64] FLAG: --register-with-taints=""
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231921    4447 flags.go:64] FLAG: --registry-burst="10"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231924    4447 flags.go:64] FLAG: --registry-qps="5"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231927    4447 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231930    4447 flags.go:64] FLAG: --reserved-memory=""
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231934    4447 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 13:08:32.240329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231937    4447 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231940    4447 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231943    4447 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231946    4447 flags.go:64] FLAG: --runonce="false"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231949    4447 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231952    4447 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231955    4447 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231958    4447 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231961    4447 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231964    4447 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231967    4447 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231970    4447 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231973    4447 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231975    4447 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231978    4447 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231981    4447 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231984    4447 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231987    4447 flags.go:64] FLAG: --system-cgroups=""
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231992    4447 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.231998    4447 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232001    4447 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232004    4447 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232007    4447 flags.go:64] FLAG: --tls-min-version=""
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232010    4447 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232013    4447 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 13:08:32.241103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232016    4447 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232019    4447 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232022    4447 flags.go:64] FLAG: --v="2"
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232026    4447 flags.go:64] FLAG: --version="false"
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232030    4447 flags.go:64] FLAG: --vmodule=""
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232034    4447 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232037    4447 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232164    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232167    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232171    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232173    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232176    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232179    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232181    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232184    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232186    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232189    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232191    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232194    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232196    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232199    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:32.241740 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232201    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232204    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232207    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232209    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232214    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232217    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232220    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232222    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232225    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232227    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232230    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232233    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232235    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232238    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232240    4447 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232243    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232245    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232248    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232250    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232253    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:32.242388 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232255    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232258    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232260    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232263    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232265    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232268    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232270    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232273    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232276    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232279    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232282    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232284    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232286    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232289    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232292    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232294    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232298    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232300    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232303    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232306    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:32.243087 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232309    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232311    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232314    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232316    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232319    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232321    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232324    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232326    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232329    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232332    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232334    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232337    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232354    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232357    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232360    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232362    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232365    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232369    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232372    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232374    4447 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:32.243723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232377    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232380    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232382    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232385    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232387    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232390    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232393    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232397    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232402    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232404    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232408    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232411    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232414    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232417    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232420    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232422    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232425    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232427    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232430    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:32.244540 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232433    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232436    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232438    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232441    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232443    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232446    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232448    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232451    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232455    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232458    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232462    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232464    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232467    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.232493    4447 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232528    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232533    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:32.245017 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232535    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232538    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232541    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232544    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232546    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232551    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232554    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232556    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232559    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232561    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232565    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232567    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232570    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232572    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232575    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232577    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232581    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232583    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232586    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232588    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:32.245436 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232591    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232594    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232596    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232599    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232601    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232604    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232606    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232611    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232614    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232616    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232619    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232621    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232624    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232627    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232630    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232633    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232635    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232639    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232642    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232644    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:32.245951 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232647    4447 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232649    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232652    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232655    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232658    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232660    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232663    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232667    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232671    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232674    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232677    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232679    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232682    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232685    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232687    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232690    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232692    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232695    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232697    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:32.246560 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232701    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232705    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232708    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232710    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232713    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232715    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232718    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232721    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232723    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232727    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232732    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232735    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232738    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232740    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232743    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232746    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232748    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232752    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232754    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232757    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:32.255072 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232759    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232762    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232764    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232767    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232770    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232772    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232775    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232777    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232780    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232782    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232785    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232787    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232791    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232794    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232796    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232799    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232802    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232804    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232808    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:32.256041 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232810    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232813    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232816    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232819    4447 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232822    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.232824    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.238080    4447 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.238097    4447 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.238119    4447 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.238124    4447 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238179    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238184    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238187    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238190    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238193    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238196    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238200    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238203    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238205    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238208    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:32.256839 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238211    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238214    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238216    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238219    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238224    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238229    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238231    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238234    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238237    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238240    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238243    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238245    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238248    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238252    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238254    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238257    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238260    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238263    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238265    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238268    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:32.257581 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238271    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238274    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238276    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238279    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238281    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238284    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238286    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238289    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238292    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238294    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238297    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238300    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238302    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238305    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238307    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238310    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238319    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238321    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238324    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:32.258153 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238326    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238329    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238332    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238335    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238337    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238361    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238365    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238368    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238370    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238373    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238376    4447 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238378    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238381    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238384    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238386    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238389    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238391    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238394    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238396    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238399    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:32.258725 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238401    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238404    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238407    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238409    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238412    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238414    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238417    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238419    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238422    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238424    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238433    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238436    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238439    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238442    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238444    4447 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238447    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238449    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238452    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238455    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:32.259420 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238457    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238460    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238462    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238465    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238468    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238471    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238473    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238476    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238479    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238482    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238484    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238487    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238489    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238492    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238496    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238500    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238504    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:32.259879 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238506    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.238534    4447 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238686    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238691    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238695    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238697    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238700    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238704    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238707    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238710    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238713    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238718    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238721    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238723    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238726    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238728    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:32.260286 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238731    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238734    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238736    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238739    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238741    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238744    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238746    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238749    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238752    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238754    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238757    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238759    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238762    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238764    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238767    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238770    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238773    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238775    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238779    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238781    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:32.260698 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238784    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238786    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238789    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238792    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238795    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238798    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238801    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238803    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238805    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238808    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238811    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238813    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238816    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238818    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238821    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238823    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238826    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238828    4447 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238843    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238847    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:32.261188 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238850    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238853    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238855    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238858    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238862    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238866    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238869    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238871    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238874    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238877    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238879    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238882    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238885    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238887    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238890    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238893    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238895    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238899    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238901    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:32.261685 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238904    4447 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238906    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238909    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238911    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238914    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238916    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238919    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238921    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238924    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238926    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238929    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238932    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238934    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238937    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238939    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238942    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238944    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238947    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238949    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:32.262165 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238952    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238954    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238957    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238960    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238962    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238965    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238967    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238970    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238972    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238975    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238977    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238980    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238983    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.238986    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.239008    4447 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239037    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:32.262675 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239041    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239044    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239047    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239050    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239053    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239056    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239059    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239062    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239065    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239068    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239070    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239073    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239075    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239078    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239081    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239085    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239089    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239094    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239097    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239100    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:32.263076 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239102    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239105    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239108    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239110    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239113    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239115    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239117    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239120    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239123    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239126    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239129    4447 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239131    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239134    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239136    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239139    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239141    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239144    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239147    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239151    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:32.263682 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239154    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239158    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239161    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239165    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239168    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239171    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239173    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239176    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239178    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239181    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239183    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239186    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239189    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239191    4447 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239194    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239196    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239198    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239201    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239203    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:32.264298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239206    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239209    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239211    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239213    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239217    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239220    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239222    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239224    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239227    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239229    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239232    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239234    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239237    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239240    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239242    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239244    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239247    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239249    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239252    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:32.264800 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239254    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239257    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239260    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239262    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239265    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239269    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239272    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239274    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239277    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239280    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239282    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239285    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239287    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239290    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239292    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239295    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239300    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239303    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239306    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239309    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:32.265258 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239312    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239314    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239317    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239319    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239322    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239324    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239327    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:32.239330    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.239399    4447 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.240363    4447 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.240413    4447 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.240590    4447 server.go:1014] "Starting client certificate rotation"
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.240737    4447 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.240809    4447 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 13:02:53 +0000 UTC" deadline="2028-07-29 10:47:00.815268442 +0000 UTC"
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.240849    4447 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15213h38m28.574423501s"
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.241113    4447 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 13:08:32.265823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.242155    4447 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 13:08:32.266445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.243844    4447 log.go:25] "Validated CRI v1 runtime API"
Nov 03 13:08:32.266445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.245645    4447 log.go:25] "Validated CRI v1 image API"
Nov 03 13:08:32.266445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.246404    4447 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 13:08:32.266445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.248009    4447 fs.go:135] Filesystem UUIDs: map[570e1c45-0b29-4daf-b785-3acf9c15b081:/dev/nvme0n1p4 7483819c-7238-4cde-a9cf-dd7d874e43a0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 03 13:08:32.266660 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.248029    4447 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d31311f07b072c3954f643b71cfeeb311b29eb278ea4cb97ac487b085ee457b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d31311f07b072c3954f643b71cfeeb311b29eb278ea4cb97ac487b085ee457b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d1571b49cb1c7df6bbc7557d742185cf9872c169a868e5c43f82e3e914809af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d1571b49cb1c7df6bbc7557d742185cf9872c169a868e5c43f82e3e914809af/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e596cffd821982e01aac8bf393a629ff7f669f1e46d898bbbfe943af8ea7574/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e596cffd821982e01aac8bf393a629ff7f669f1e46d898bbbfe943af8ea7574/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e7c3e44d06308063cee8fa279e17a7654386af7e2096777d3cff63364e7901b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e7c3e44d06308063cee8fa279e17a7654386af7e2096777d3cff63364e7901b/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f36c5f908a0e3ecfd15dfb0c1b32d37e42eaaccb99fe1e7e505befb895958d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f36c5f908a0e3ecfd15dfb0c1b32d37e42eaaccb99fe1e7e505befb895958d7/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d99166880f3ba57c3dd45e928f430c8e4c88dc84c2e7713c96f8f3b4e9208eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d99166880f3ba57c3dd45e928f430c8e4c88dc84c2e7713c96f8f3b4e9208eb/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74156cec001d800de4637071ad0f1fa24e2707746dcd54ebeac0c88a9b40a298/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74156cec001d800de4637071ad0f1fa24e2707746dcd54ebeac0c88a9b40a298/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a224f8518796cc0c15709c56f6578d9c820474c1527c6ec4d12439b0b6ccb1ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a224f8518796cc0c15709c56f6578d9c820474c1527c6ec4d12439b0b6ccb1ec/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7e20460e2b520467b392129ef9ef9d2cc4f478329e0a431d39313a958ac7224/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7e20460e2b520467b392129ef9ef9d2cc4f478329e0a431d39313a958ac7224/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad0cdcc8dcdd9f4f7a79ab7de95b4f095710ca0b7e17481f2173ec2db307ee03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad0cdcc8dcdd9f4f7a79ab7de95b4f095710ca0b7e17481f2173ec2db307ee03/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b08bac6b69c078563f5cd4962940ec2ddd902be2e677e3ca4cfc212fe2dfbd04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b08bac6b69c078563f5cd4962940ec2ddd902be2e677e3ca4cfc212fe2dfbd04/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b17899e6d9fb087950134182a897dac8c7a3c3d3c7d29562cdfb2c9bdb57f62f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b17899e6d9fb087950134182a897dac8c7a3c3d3c7d29562cdfb2c9bdb57f62f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d92b628d953dcfd79bbb5c3ca497b658f08a7e262d5ac583c4e8f6ed6e9c1d8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d92b628d953dcfd79bbb5c3ca497b658f08a7e262d5ac583c4e8f6ed6e9c1d8c/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de6a626e38afd4e614b20f6bbb5dd5955d5a422ac21f07ef66539a93e7857232/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de6a626e38afd4e614b20f6bbb5dd5955d5a422ac21f07ef66539a93e7857232/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e993aaafb8602fa337c3e113c05717357b281013437acf7c97292ba70a4e011d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e993aaafb8602fa337c3e113c05717357b281013437acf7c97292ba70a4e011d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ef0c534-57ad-4269-8d12-e2a5879a3eed/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0ef0c534-57ad-4269-8d12-e2a5879a3eed/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b6197ff-1085-4ade-a095-688ff39c5f9c/volumes/kubernetes.io~projected/kube-api-access-vflt4:{mountpoint:/var/lib/kubelet/pods/1b6197ff-1085-4ade-a095-688ff39c5f9c/volumes/kubernetes.io~projected/kube-api-access-vflt4 major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aaa0065-2805-432d-8c2e-ef03591b6c90/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2aaa0065-2805-432d-8c2e-ef03591b6c90/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/378ce7f3-c33f-44f5-a383-a8acc714f547/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/378ce7f3-c33f-44f5-a383-a8acc714f547/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/378ce7f3-c33f-44f5-a383-a8acc714f547/volumes/kubernetes.io~projected/kube-api-access-t69vq:{mountpoint:/var/lib/kubelet/pods/378ce7f3-c33f-44f5-a383-a8acc714f547/volumes/kubernetes.io~projected/kube-api-access-t69vq major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/378ce7f3-c33f-44f5-a383-a8acc714f547/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/378ce7f3-c33f-44f5-a383-a8acc714f547/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f/volumes/kubernetes.io~projected/kube-api-access-4xwvb:{mountpoint:/var/lib/kubelet/pods/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f/volumes/kubernetes.io~projected/kube-api-access-4xwvb major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44f5fcad-a6fb-4bd2-99aa-f4973415aafa/volumes/kubernetes.io~projected/kube-api-access-lrmwl:{mountpoint:/var/lib/kubelet/pods/44f5fcad-a6fb-4bd2-99aa-f4973415aafa/volumes/kubernetes.io~projected/kube-api-access-lrmwl major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4da03b13-3a30-48de-b399-ba82b071c456/volumes/kubernetes.io~projected/kube-api-access-mg2kd:{mountpoint:/var/lib/kubelet/pods/4da03b13-3a30-48de-b399-ba82b071c456/volumes/kubernetes.io~projected/kube-api-access-mg2kd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12/volumes/kubernetes.io~projected/kube-api-access-n6bhd:{mountpoint:/var/lib/kubelet/pods/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12/volumes/kubernetes.io~projected/kube-api-access-n6bhd major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69042048-b2e6-477d-a330-94d2a1e608e3/volumes/kubernetes.io~projected/kube-api-access-nw6jc:{mountpoint:/var/lib/kubelet/pods/69042048-b2e6-477d-a330-94d2a1e608e3/volumes/kubernetes.io~projected/kube-api-access-nw6jc major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70f3bdac-e826-4187-b83c-993fe2c5d7ac/volumes/kubernetes.io~projected/kube-api-access-w9rrw:{mountpoint:/var/lib/kubelet/pods/70f3bdac-e826-4187-b83c-993fe2c5d7ac/volumes/kubernetes.io~projected/kube-api-access-w9rrw major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2e2e850-03eb-485d-940e-1b6d9063091c/volumes/kubernetes.io~projected/kube-api-access-4czgz:{mountpoint:/var/lib/kubelet/pods/b2e2e850-03eb-485d-940e-1b6d9063091c/volumes/kubernetes.io~projected/kube-api-access-4czgz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4100403-c8f8-4ac1-98c4-2795d5a11b10/volumes/kubernetes.io~projected/kube-api-access-vrblj:{mountpoint:/var/lib/kubelet/pods/b4100403-c8f8-4ac1-98c4-2795d5a11b10/volumes/kubernetes.io~projected/kube-api-access-vrblj major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6257887-6b6e-4ad3-ae76-3d121659e2d4/volumes/kubernetes.io~projected/kube-api-access-shg67:{mountpoint:/var/lib/kubelet/pods/b6257887-6b6e-4ad3-ae76-3d121659e2d4/volumes/kubernetes.io~projected/kube-api-access-shg67 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba1ed75b-f38b-4589-87fa-0c8d22abe784/volumes/kubernetes.io~projected/kube-api-access-l9k4v:{mountpoint:/var/lib/kubelet/pods/ba1ed75b-f38b-4589-87fa-0c8d22abe784/volumes/kubernetes.io~projected/kube-api-access-l9k4v major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d237eda3-f867-4dd3-8a9b-24aaa9c37403/volumes/kubernetes.io~projected/kube-api-access-tqb6m:{mountpoint:/var/lib/kubelet/pods/d237eda3-f867-4dd3-8a9b-24aaa9c37403/volumes/kubernetes.io~projected/kube-api-access-tqb6m major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d58899f4-efb6-4d21-b1eb-fcea8c046356/volumes/kubernetes.io~projected/kube-api-access-cpjw8:{mountpoint:/var/lib/kubelet/pods/d58899f4-efb6-4d21-b1eb-fcea8c046356/volumes/kubernetes.io~projected/kube-api-access-cpjw8 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41/volumes/kubernetes.io~projected/kube-api-access-llfvj:{mountpoint:/var/lib/kubelet/pods/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41/volumes/kubernetes.io~projected/kube-api-access-llfvj major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/698a8d8b004f51378f9b9336674b3e40e4111d5f650bb762ea7ee016f9a6ff77/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3766c4e543192f3fe31a340b74c45a381eeacc12bf64f0786d3f59dc810bada7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/821fcde57bbf206fae343f0a206e402fdbd901eebf47828c4b425763d8212627/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f68e65b0bcbdc4023e99d699ced4525e92a54b032b413d16d5f124832e6feda5/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4fbd575f2020125252650a5f54eda20e045ac03b5bc02f040d8aeabbe7300dd8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a6992fb718083618622dbd1baa5ef6517c2fabbfd24d4ab7ef5abdd1a8cc1d4b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e6d889a79db32120de0df12b01324c32f1f66b3418f891a61450a9ff57d3f2de/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/43093b477542cf4b16f9630aa7b820378f4929fb733a405d5396eaaccd20f528/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0fe85ed6f215ce78fea833c3853856eeee8c9efdb198bef84a8f554b298eec67/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b47ab74203afbae092e63754018ff0692008b524c0ff3ba99e2614c7d4dd261e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/05ee99d90ab5521b9920e89fa1db75d0f360e5684ccb49018fa4d966d5d3ff36/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/44f43b8552cb22dccd440861227c279447b239072c25f0891763fe91d4b2e4c3/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/47091e02837034372aebbcf5b46ac30d6d7f529c1b5f9b9a925021a85f9e21eb/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/49bf4a6a5a8f4a02ae5afc22c9a631a3a7764e1fcea5a21eb15c02f0bb17e709/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/76a2ba8f0f190bdbf149a5a3d7f3fe843482e72437d042a58e633cda666fe87b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d769ab70a1b8404fb4d662dcc8c3a4ce7ae9aabbcda05c179bbaaf228fa71d7d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a5a90a6014b2778ca528620ebbdd537b94266a08e75352fab223d8a4371cc8a8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6e10973c0e59709f62c5a231917e328e1297519a9925a55dd2f0a29ee56ef364/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/72f2f13bad928d177c8689ad3183484242931c946e0e400ec0139f4a1127146a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/93ceab79a3ce0614f61f384aee1d6a23e6fa9fc418c39331d65d639f4da8a1f3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1a08f2f682334bb7f205147a7b75117011f6fc8d4f4e560d2787399f721decf9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/83eddd4c617360c61d8c47c96a79864c8fae4751b10fb54a637a012493548203/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/254d306201961b39b38a484883e240c3496547d7317fff35661b17b7cde84485/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/842a911f17e250209f62e9adddcc3eeb9de5b111d88d7f62ac79882c73a64bfb/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/5cd51e11280de317d0af252d93f68ac715c2961d871b08f4e772650c28cbf2ea/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/82bf1f49b419ca4a63950a8dfb1527106833360351866f1874bbb13cf7c9a38e/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/28c9764fedb4496f0b2df7b94eb28f355f4c595bddf356f38db154a54af4879a/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/b6cfbd35a22f645532984227dc9e17a83dbe11c932e5acd449a4539dfb02db81/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/5e6128bf5eccbeced9c0968b0e6d2d8c5bf4e1b641cca3e03a6e56ccd1f8f447/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/a1f68a7a3152e1424eefd0deb1c9e9f577bd6f6be684ec2e9b1cd4af99240dd5/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/4e3f9df40b12f41358c0ea96405352c45564d29cb9e7d8111898f7e626c01126/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/31ab3632b435f77b103fc1db699e7cfcaa8356d5825621f4f376cd246d7d68ea/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/93c075244e55c1c9c5a649b2961b832c9b3f83f1e70f436b6182b2490e3d8a9a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6802799dc5dc91239eafc4ad077cb5eee95abd2a5f3bb81cfa8ac251d3fd29ed/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bb9c00d0a8aecc9e9dce6033f56fa34626f98f7b802d05df094e0ef3c2dd99e3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/89f743380d4398d9a637f8f8ac1aeb6b7f3828f79bf63640116baaf48562343d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/33cbb5013f0427e37e7e9fb4da9678da0ca53aa9be7c9a00593ef0e48022c04e/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/92f5e6c2e1cad83c040a0fa1690daa002910bde0aafea7908a62f4d48e123a0b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/47d4d4f8ec4304d57c192a72c42defcb76adfbbbb3d6c4a61b507247d839ddb0/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/863ffda174482241f4460831c39da4449250d3afc4f3fd204c0314c40e88a186/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.256554    4447 manager.go:217] Machine: {Timestamp:2026-11-03 13:08:32.255733419 +0000 UTC m=+0.067401600 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099875 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec235da851e4140fee3d94aff9926027 SystemUUID:ec235da8-51e4-140f-ee3d-94aff9926027 BootID:b1f81186-ba52-4625-b5be-d04a097307e7 Filesystems:[{Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d237eda3-f867-4dd3-8a9b-24aaa9c37403/volumes/kubernetes.io~projected/kube-api-access-tqb6m DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f/volumes/kubernetes.io~projected/kube-api-access-4xwvb DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ef0c534-57ad-4269-8d12-e2a5879a3eed/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f36c5f908a0e3ecfd15dfb0c1b32d37e42eaaccb99fe1e7e505befb895958d7/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74156cec001d800de4637071ad0f1fa24e2707746dcd54ebeac0c88a9b40a298/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1b6197ff-1085-4ade-a095-688ff39c5f9c/volumes/kubernetes.io~projected/kube-api-access-vflt4 DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4da03b13-3a30-48de-b399-ba82b071c456/volumes/kubernetes.io~projected/kube-api-access-mg2kd DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e596cffd821982e01aac8bf393a629ff7f669f1e46d898bbbfe943af8ea7574/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b08bac6b69c078563f5cd4962940ec2ddd902be2e677e3ca4cfc212fe2dfbd04/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41/volumes/kubernetes.io~projected/kube-api-access-llfvj DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d1571b49cb1c7df6bbc7557d742185cf9872c169a868e5c43f82e3e914809af/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69042048-b2e6-477d-a330-94d2a1e608e3/volumes/kubernetes.io~projected/kube-api-access-nw6jc DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/378ce7f3-c33f-44f5-a383-a8acc714f547/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7e20460e2b520467b392129ef9ef9d2cc4f478329e0a431d39313a958ac7224/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a224f8518796cc0c15709c56f6578d9c820474c1527c6ec4d12439b0b6ccb1ec/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/44f5fcad-a6fb-4bd2-99aa-f4973415aafa/volumes/kubernetes.io~projected/kube-api-access-lrmwl DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad0cdcc8dcdd9f4f7a79ab7de95b4f095710ca0b7e17481f2173ec2db307ee03/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4100403-c8f8-4ac1-98c4-2795d5a11b10/volumes/kubernetes.io~projected/kube-api-access-vrblj DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de6a626e38afd4e614b20f6bbb5dd5955d5a422ac21f07ef66539a93e7857232/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d99166880f3ba57c3dd45e928f430c8e4c88dc84c2e7713c96f8f3b4e9208eb/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/378ce7f3-c33f-44f5-a383-a8acc714f547/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b6257887-6b6e-4ad3-ae76-3d121659e2d4/volumes/kubernetes.io~projected/kube-api-access-shg67 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d58899f4-efb6-4d21-b1eb-fcea8c046356/volumes/kubernetes.io~projected/kube-api-access-cpjw8 DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b17899e6d9fb087950134182a897dac8c7a3c3d3c7d29562cdfb2c9bdb57f62f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba1ed75b-f38b-4589-87fa-0c8d22abe784/volumes/kubernetes.io~projected/kube-api-access-l9k4v DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e7c3e44d06308063cee8fa279e17a7654386af7e2096777d3cff63364e7901b/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2aaa0065-2805-432d-8c2e-ef03591b6c90/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12/volumes/kubernetes.io~projected/kube-api-access-n6bhd DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/378ce7f3-c33f-44f5-a383-a8acc714f547/volumes/kubernetes.io~projected/kube-api-access-t69vq DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2e2e850-03eb-485d-940e-1b6d9063091c/volumes/kubernetes.io~projected/kube-api-access-4czgz DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e993aaafb8602fa337c3e113c05717357b281013437acf7c97292ba70a4e011d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d92b628d953dcfd79bbb5c3ca497b658f08a7e262d5ac583c4e8f6ed6e9c1d8c/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/70f3bdac-e826-4187-b83c-993fe2c5d7ac/volumes/kubernetes.io~projected/kube-api-access-w9rrw DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d31311f07b072c3954f643b71cfeeb311b29eb278ea4cb97ac487b085ee457b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:5e7c3e44d063080 MacAddress:e2:dd:c5:ed:c8:41 Speed:10000 Mtu:8901} {Name:6d99166880f3ba5 MacAddress:ea:60:c3:37:81:76 Speed:10000 Mtu:8901} {Name:a7e20460e2b5204 MacAddress:3e:65:b9:4e:2a:7a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ea:ac:87:71 Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:a9:8b:43:be:dd Speed:0 Mtu:8901} {Name:d92b628d953dcfd MacAddress:9e:73:b0:71:6b:7b Speed:10000 Mtu:8901} {Name:de6a626e38afd4e MacAddress:e2:9f:d0:05:bb:de Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ea:ac:87:71 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:a6:8a:f5:4e:31:9b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:2d:e8:aa:7f:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257027    4447 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257101    4447 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257401    4447 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257422    4447 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-167.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257557    4447 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257565    4447 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257576    4447 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257596    4447 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257687    4447 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257843    4447 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257942    4447 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257974    4447 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257984    4447 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.257993    4447 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.258003    4447 kubelet.go:405] "Adding apiserver pod source"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.258011    4447 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.258646    4447 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.258656    4447 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 13:08:32.266916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.259197    4447 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 13:08:32.269948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.269931    4447 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 13:08:32.269948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.269946    4447 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 13:08:32.270071 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270051    4447 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 13:08:32.270244 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270232    4447 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 13:08:32.270287 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270262    4447 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 13:08:32.270399 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270388    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 13:08:32.270436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270405    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 13:08:32.270436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270411    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 13:08:32.270436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270419    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 13:08:32.270436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270428    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 13:08:32.270436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270437    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 13:08:32.270579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270443    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 13:08:32.270579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270449    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 13:08:32.270579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270456    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 13:08:32.270579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270462    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 13:08:32.270579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270471    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 13:08:32.270579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270481    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 13:08:32.270579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270509    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 13:08:32.270579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.270517    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 13:08:32.271639 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.271622    4447 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:32.272519 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.272502    4447 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:32.274529 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.274517    4447 server.go:1263] "Started kubelet"
Nov 03 13:08:32.274730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.274617    4447 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 13:08:32.275542 ip-10-0-132-167 systemd[1]: Started Kubernetes Kubelet.
Nov 03 13:08:32.279544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.274656    4447 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 13:08:32.279544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.276358    4447 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 13:08:32.279544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.276675    4447 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 13:08:32.279544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.277957    4447 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 13:08:32.279828 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.279813    4447 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 13:08:32.280193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.280177    4447 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 13:08:32.280268 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.280231    4447 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 13:08:32.280268 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.280258    4447 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 13:08:32.280376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.280339    4447 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 13:08:32.280473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.280434    4447 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 13:02:55 +0000 UTC" deadline="2028-05-28 03:37:07.068130541 +0000 UTC"
Nov 03 13:08:32.280538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.280475    4447 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13718h28m34.787658948s"
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282591    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70f3bdac-e826-4187-b83c-993fe2c5d7ac" volumeName="kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282656    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4100403-c8f8-4ac1-98c4-2795d5a11b10" volumeName="kubernetes.io/configmap/b4100403-c8f8-4ac1-98c4-2795d5a11b10-config-volume" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282678    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ef0c534-57ad-4269-8d12-e2a5879a3eed" volumeName="kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282693    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aaa0065-2805-432d-8c2e-ef03591b6c90" volumeName="kubernetes.io/secret/2aaa0065-2805-432d-8c2e-ef03591b6c90-agent-certs" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282705    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="378ce7f3-c33f-44f5-a383-a8acc714f547" volumeName="kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282717    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" volumeName="kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282728    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="378ce7f3-c33f-44f5-a383-a8acc714f547" volumeName="kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-config" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282739    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f5fcad-a6fb-4bd2-99aa-f4973415aafa" volumeName="kubernetes.io/projected/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-kube-api-access-lrmwl" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282754    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58899f4-efb6-4d21-b1eb-fcea8c046356" volumeName="kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282766    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" volumeName="kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282779    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="378ce7f3-c33f-44f5-a383-a8acc714f547" volumeName="kubernetes.io/secret/378ce7f3-c33f-44f5-a383-a8acc714f547-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282791    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4da03b13-3a30-48de-b399-ba82b071c456" volumeName="kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-binary-copy" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282809    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12" volumeName="kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-tuned" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282821    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0bc4a29-7ff7-4b22-8f9f-56906b70ee41" volumeName="kubernetes.io/secret/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-cert" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282836    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="378ce7f3-c33f-44f5-a383-a8acc714f547" volumeName="kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-env-overrides" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282849    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" volumeName="kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282861    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f5fcad-a6fb-4bd2-99aa-f4973415aafa" volumeName="kubernetes.io/empty-dir/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-tmp-dir" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282873    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4da03b13-3a30-48de-b399-ba82b071c456" volumeName="kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282898    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4100403-c8f8-4ac1-98c4-2795d5a11b10" volumeName="kubernetes.io/projected/b4100403-c8f8-4ac1-98c4-2795d5a11b10-kube-api-access-vrblj" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282909    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6257887-6b6e-4ad3-ae76-3d121659e2d4" volumeName="kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cni-binary-copy" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282921    4447 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282936    4447 factory.go:55] Registering systemd factory
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282950    4447 factory.go:223] Registration of the systemd container factory successfully
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.282922    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba1ed75b-f38b-4589-87fa-0c8d22abe784" volumeName="kubernetes.io/configmap/ba1ed75b-f38b-4589-87fa-0c8d22abe784-serviceca" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283029    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d237eda3-f867-4dd3-8a9b-24aaa9c37403" volumeName="kubernetes.io/projected/d237eda3-f867-4dd3-8a9b-24aaa9c37403-kube-api-access-tqb6m" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283052    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12" volumeName="kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-tmp" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283068    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12" volumeName="kubernetes.io/projected/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-kube-api-access-n6bhd" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283081    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69042048-b2e6-477d-a330-94d2a1e608e3" volumeName="kubernetes.io/projected/69042048-b2e6-477d-a330-94d2a1e608e3-kube-api-access-nw6jc" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283096    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b6197ff-1085-4ade-a095-688ff39c5f9c" volumeName="kubernetes.io/projected/1b6197ff-1085-4ade-a095-688ff39c5f9c-kube-api-access-vflt4" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283111    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aaa0065-2805-432d-8c2e-ef03591b6c90" volumeName="kubernetes.io/configmap/2aaa0065-2805-432d-8c2e-ef03591b6c90-konnectivity-ca" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283124    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" volumeName="kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283143    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4da03b13-3a30-48de-b399-ba82b071c456" volumeName="kubernetes.io/projected/4da03b13-3a30-48de-b399-ba82b071c456-kube-api-access-mg2kd" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283158    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58899f4-efb6-4d21-b1eb-fcea8c046356" volumeName="kubernetes.io/projected/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-api-access-cpjw8" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283171    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="378ce7f3-c33f-44f5-a383-a8acc714f547" volumeName="kubernetes.io/projected/378ce7f3-c33f-44f5-a383-a8acc714f547-kube-api-access-t69vq" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283213    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6257887-6b6e-4ad3-ae76-3d121659e2d4" volumeName="kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-daemon-config" seLinuxMountContext=""
Nov 03 13:08:32.283193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283228    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6257887-6b6e-4ad3-ae76-3d121659e2d4" volumeName="kubernetes.io/projected/b6257887-6b6e-4ad3-ae76-3d121659e2d4-kube-api-access-shg67" seLinuxMountContext=""
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283245    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58899f4-efb6-4d21-b1eb-fcea8c046356" volumeName="kubernetes.io/empty-dir/d58899f4-efb6-4d21-b1eb-fcea8c046356-data-volume" seLinuxMountContext=""
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283259    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0bc4a29-7ff7-4b22-8f9f-56906b70ee41" volumeName="kubernetes.io/projected/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-kube-api-access-llfvj" seLinuxMountContext=""
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283272    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba1ed75b-f38b-4589-87fa-0c8d22abe784" volumeName="kubernetes.io/projected/ba1ed75b-f38b-4589-87fa-0c8d22abe784-kube-api-access-l9k4v" seLinuxMountContext=""
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283284    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4da03b13-3a30-48de-b399-ba82b071c456" volumeName="kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283298    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e2e850-03eb-485d-940e-1b6d9063091c" volumeName="kubernetes.io/configmap/b2e2e850-03eb-485d-940e-1b6d9063091c-iptables-alerter-script" seLinuxMountContext=""
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283311    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e2e850-03eb-485d-940e-1b6d9063091c" volumeName="kubernetes.io/projected/b2e2e850-03eb-485d-940e-1b6d9063091c-kube-api-access-4czgz" seLinuxMountContext=""
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283326    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4100403-c8f8-4ac1-98c4-2795d5a11b10" volumeName="kubernetes.io/empty-dir/b4100403-c8f8-4ac1-98c4-2795d5a11b10-tmp-dir" seLinuxMountContext=""
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283338    4447 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283361    4447 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283285    4447 factory.go:153] Registering CRI-O factory
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283492    4447 factory.go:223] Registration of the crio container factory successfully
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283525    4447 factory.go:103] Registering Raw factory
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283543    4447 manager.go:1199] Started watching for new ooms in manager
Nov 03 13:08:32.284961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.283968    4447 manager.go:319] Starting recovery of all containers
Nov 03 13:08:32.287105 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.287077    4447 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:32.287190 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.287149    4447 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 13:08:32.310334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.310285    4447 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 13:08:32.311539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.311515    4447 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 13:08:32.311539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.311538    4447 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 13:08:32.311699 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.311564    4447 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 13:08:32.311699 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:32.311606    4447 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 13:08:32.314401 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.314379    4447 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:32.322970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.322935    4447 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="a13cdc30eadcd5e2b06d6053b692401cf49aec9d07c8a9cb2489c4fde2858fb9" exitCode=0
Nov 03 13:08:32.322970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.322965    4447 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="30bc73e0c269e19f3f427c70243e2ed6cc5ffac962f566b9607baf6ea3e7004f" exitCode=0
Nov 03 13:08:32.322970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.322971    4447 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="d24ffd5a9e79619a78a0b70edb7a3a73ff9dc6c2b1e419c93e17104267cb69c3" exitCode=0
Nov 03 13:08:32.322970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.322977    4447 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="7b4a691a7a3c68c24da9250e660e31e21e384beef09d6bf716c49970056ca5c7" exitCode=0
Nov 03 13:08:32.323248 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.322983    4447 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="853a68e1545e888f2bb55c609f8d8a52b1570b8ccf656f1198b34ff9b750f38b" exitCode=0
Nov 03 13:08:32.323248 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.322989    4447 generic.go:356] "Generic (PLEG): container finished" podID="4da03b13-3a30-48de-b399-ba82b071c456" containerID="1d2f5a33df48981cf2aa09979a1c1c02307eb0be606b113d2b5a567f600c7c4a" exitCode=0
Nov 03 13:08:32.345686 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.345650    4447 generic.go:356] "Generic (PLEG): container finished" podID="bba9b1d6fba3e1fb24d7e6543c829290" containerID="a49a9057c0dddeda639211f58c4e5ff2f85c73164fd32c20816c40acf6ecc1db" exitCode=0
Nov 03 13:08:32.354091 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.354068    4447 manager.go:324] Recovery completed
Nov 03 13:08:32.368582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368534    4447 cpu_manager.go:226] "Starting" policy="none"
Nov 03 13:08:32.368582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368553    4447 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 13:08:32.368582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368575    4447 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 13:08:32.368846 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368726    4447 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 13:08:32.368846 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368741    4447 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 13:08:32.368846 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368765    4447 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 13:08:32.368846 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368773    4447 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 13:08:32.368846 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368781    4447 policy_none.go:50] "Start"
Nov 03 13:08:32.368846 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368793    4447 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 13:08:32.368846 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368806    4447 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 13:08:32.369160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.368991    4447 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 13:08:32.369160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.369002    4447 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 13:08:32.369160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.369011    4447 policy_none.go:44] "Start"
Nov 03 13:08:32.373909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.373218    4447 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 13:08:32.378609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.378596    4447 manager.go:341] "Starting Device Plugin manager"
Nov 03 13:08:32.378661 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:32.378623    4447 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 13:08:32.378661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.378634    4447 server.go:87] "Starting device plugin registration server"
Nov 03 13:08:32.378913 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.378899    4447 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 13:08:32.378972 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.378911    4447 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 13:08:32.379016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.378983    4447 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 13:08:32.379175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.379107    4447 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 13:08:32.379252 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.379182    4447 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 13:08:32.379252 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.379190    4447 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 13:08:32.379566 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.379470    4447 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:08:32.379167863Z","UUID":"eea0126d-b44b-41a2-825d-845b6952ce4d","Handler":null,"Name":"","Endpoint":""}
Nov 03 13:08:32.380522 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:32.380502    4447 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 13:08:32.380999 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.380980    4447 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 13:08:32.381088 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.381008    4447 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 13:08:32.412548 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.412454    4447 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"]
Nov 03 13:08:32.412979 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.412950    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"6c4c57a57b1c5fc6c67e60988666a273","Type":"ContainerStarted","Data":"ecb0265ba4916a75056aa397d5216c66ea3435be0ff769038c8380f541439858"}
Nov 03 13:08:32.413017 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.412989    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"6c4c57a57b1c5fc6c67e60988666a273","Type":"ContainerStarted","Data":"a224f8518796cc0c15709c56f6578d9c820474c1527c6ec4d12439b0b6ccb1ec"}
Nov 03 13:08:32.413068 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.413055    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"bba9b1d6fba3e1fb24d7e6543c829290","Type":"ContainerStarted","Data":"f5b44ff0d716cdce2a0136f69d33cdb8d1f5e686057d184b0dd1b484bbc84d2c"}
Nov 03 13:08:32.413104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.413078    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"bba9b1d6fba3e1fb24d7e6543c829290","Type":"ContainerDied","Data":"a49a9057c0dddeda639211f58c4e5ff2f85c73164fd32c20816c40acf6ecc1db"}
Nov 03 13:08:32.413104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.413091    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal" event={"ID":"bba9b1d6fba3e1fb24d7e6543c829290","Type":"ContainerStarted","Data":"b17899e6d9fb087950134182a897dac8c7a3c3d3c7d29562cdfb2c9bdb57f62f"}
Nov 03 13:08:32.413104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.413058    4447 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.413187 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.413152    4447 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.429387 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.429368    4447 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:08:32.429466 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:32.429418    4447 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.429693 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.429681    4447 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:08:32.429726 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:32.429710    4447 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.480832 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.480793    4447 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:08:32.482164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.482136    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:08:32.482276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.482175    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:08:32.482276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.482190    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:08:32.482276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.482234    4447 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.487623 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.487604    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.487694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.487631    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.487694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.487659    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c4c57a57b1c5fc6c67e60988666a273-config\") pod \"kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"6c4c57a57b1c5fc6c67e60988666a273\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.492446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.492430    4447 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.492536 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.492498    4447 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.588525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.588483    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c4c57a57b1c5fc6c67e60988666a273-config\") pod \"kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"6c4c57a57b1c5fc6c67e60988666a273\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.588525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.588506    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c4c57a57b1c5fc6c67e60988666a273-config\") pod \"kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"6c4c57a57b1c5fc6c67e60988666a273\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.588738 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.588585    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.588738 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.588614    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.588738 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.588664    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:32.588738 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:32.588665    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bba9b1d6fba3e1fb24d7e6543c829290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" (UID: \"bba9b1d6fba3e1fb24d7e6543c829290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:33.258727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.258692    4447 apiserver.go:52] "Watching apiserver"
Nov 03 13:08:33.310547 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.310511    4447 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 13:08:33.311866 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.311834    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-62dpn","openshift-service-ca/service-ca-5fffc97bdf-vmxp7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf","openshift-dns/node-resolver-fgqkh","openshift-image-registry/node-ca-88kk5","openshift-multus/network-metrics-daemon-2m9tv","openshift-network-diagnostics/network-check-target-ch6h8","kube-system/konnectivity-agent-d9722","kube-system/global-pull-secret-syncer-9wrnl","openshift-insights/insights-runtime-extractor-dqvjq","openshift-insights/periodic-gathering-f9xld-h6jgs","openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5","openshift-dns/dns-default-v5xss","openshift-multus/multus-additional-cni-plugins-gjtf7","kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal","openshift-multus/multus-mkzmk","openshift-ingress-canary/ingress-canary-x7hff","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-btbj2","openshift-network-operator/iptables-alerter-2fwt9"]
Nov 03 13:08:33.314178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.314152    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:33.316637 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.316617    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:33.317458 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.317438    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-62cxl\""
Nov 03 13:08:33.318087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.318070    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.318939 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.318916    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-569wf\""
Nov 03 13:08:33.319163 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.319144    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-knm89\""
Nov 03 13:08:33.319230 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.319204    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jqlk5\""
Nov 03 13:08:33.319410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.319393    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mrd6\""
Nov 03 13:08:33.319410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.319404    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 13:08:33.319512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.319457    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d95tp\""
Nov 03 13:08:33.319512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.319465    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c596b\""
Nov 03 13:08:33.320207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.320191    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wv2nv\""
Nov 03 13:08:33.320254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.320212    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vwt8x\""
Nov 03 13:08:33.320486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.320472    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-psqgf\""
Nov 03 13:08:33.320680 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.320664    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 13:08:33.321492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.321477    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 13:08:33.321865 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.321839    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 13:08:33.322088 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322072    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 13:08:33.322189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322174    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 13:08:33.322256 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322243    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 13:08:33.322305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322292    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wd8kt\""
Nov 03 13:08:33.322367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322297    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hltl5\""
Nov 03 13:08:33.322367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322315    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-sjjkt\""
Nov 03 13:08:33.322440 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322421    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gkggt\""
Nov 03 13:08:33.322556 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322538    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zrpxj\""
Nov 03 13:08:33.322749 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322737    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s2gms\""
Nov 03 13:08:33.322787 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.322748    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 13:08:33.323701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.323689    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s4cl2\""
Nov 03 13:08:33.325013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.324972    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.325013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.324984    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 13:08:33.325013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325011    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.325356 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325326    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 13:08:33.325420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325377    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 13:08:33.325420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325406    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.325525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325498    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.325579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325522    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 13:08:33.325628 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325576    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.325628 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325588    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 13:08:33.325709 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325679    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 13:08:33.325809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325791    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.325909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325823    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.325909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325848    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.325909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.325855    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.326044 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326032    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.326091 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326077    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.326136 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326104    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.326172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326140    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 13:08:33.326172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326146    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 13:08:33.326172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326167    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 13:08:33.326375 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326170    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.326633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326616    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.326819 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326805    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.326941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326821    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 13:08:33.326941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326828    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.327080 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326846    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 13:08:33.327080 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.326859    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.327329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.327306    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 13:08:33.327519 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.327502    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.327598 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.327584    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.327735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.327721    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.327788 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.327748    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 13:08:33.327906 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.327892    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.328614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.328587    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 13:08:33.328699 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.328683    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.347698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.347673    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.348603 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.348580    4447 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:33.348701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.348687    4447 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:33.358192 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.358172    4447 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:08:33.358281 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.358206    4447 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:08:33.358281 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:33.358227    4447 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:33.358281 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:33.358239    4447 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal"
Nov 03 13:08:33.382055 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.382028    4447 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 13:08:33.392445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392420    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.392510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392455    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-socket-dir-parent\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.392510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392477    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:33.392510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392493    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aaa0065-2805-432d-8c2e-ef03591b6c90-agent-certs\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:08:33.392608 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392555    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-hosts-file\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:08:33.392641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392614    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2e2e850-03eb-485d-940e-1b6d9063091c-host-slash\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:08:33.392727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392708    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-kubelet-config\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:33.392798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392743    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.392798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392764    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-systemd-units\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.392855 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392799    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-slash\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.392855 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392814    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aaa0065-2805-432d-8c2e-ef03591b6c90-agent-certs\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:08:33.392855 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392819    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-etc-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.392855 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392835    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrblj\" (UniqueName: \"kubernetes.io/projected/b4100403-c8f8-4ac1-98c4-2795d5a11b10-kube-api-access-vrblj\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:33.392966 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392852    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ef0c534-57ad-4269-8d12-e2a5879a3eed-original-pull-secret\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:33.392966 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392881    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-system-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.392966 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392909    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czgz\" (UniqueName: \"kubernetes.io/projected/b2e2e850-03eb-485d-940e-1b6d9063091c-kube-api-access-4czgz\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:08:33.392966 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392926    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:33.393082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.392970    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-script-lib\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.393082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393016    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-k8s-cni-cncf-io\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.393138 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393103    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-netns\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.393138 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393132    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-daemon-config\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.393193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393178    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-modprobe-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.393227 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393204    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-script-lib\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.393273 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393258    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysconfig\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.393302 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393280    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.393330 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393310    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.393389 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393371    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9k4v\" (UniqueName: \"kubernetes.io/projected/ba1ed75b-f38b-4589-87fa-0c8d22abe784-kube-api-access-l9k4v\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:08:33.393455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393443    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-daemon-config\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.393486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393463    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t69vq\" (UniqueName: \"kubernetes.io/projected/378ce7f3-c33f-44f5-a383-a8acc714f547-kube-api-access-t69vq\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.393581 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393561    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-kubelet\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.393620 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393583    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.393680 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393665    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:33.393715 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393701    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d58899f4-efb6-4d21-b1eb-fcea8c046356-crio-socket\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.393751 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393736    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:33.393875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393831    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmwl\" (UniqueName: \"kubernetes.io/projected/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-kube-api-access-lrmwl\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:08:33.393931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393884    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-system-cni-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.393931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393902    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-cnibin\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.394031 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393922    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-hostroot\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.394031 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.393967    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-etc-kubernetes\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.394031 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394016    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.394167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394057    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-systemd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.394167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394106    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-run\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.394167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394134    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-sys\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.394311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394174    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-config\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.394311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394218    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-registration-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.394311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394245    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa0065-2805-432d-8c2e-ef03591b6c90-konnectivity-ca\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:08:33.394311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394268    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-tmp-dir\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394305    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-kubernetes\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394333    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba1ed75b-f38b-4589-87fa-0c8d22abe784-host\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394372    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-tmp-dir\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394396    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-ovnkube-config\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394402    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394437    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-env-overrides\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394457    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xwvb\" (UniqueName: \"kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394467    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa0065-2805-432d-8c2e-ef03591b6c90-konnectivity-ca\") pod \"konnectivity-agent-d9722\" (UID: \"2aaa0065-2805-432d-8c2e-ef03591b6c90\") " pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394495    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58899f4-efb6-4d21-b1eb-fcea8c046356-data-volume\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394511    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2e2e850-03eb-485d-940e-1b6d9063091c-iptables-alerter-script\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:08:33.394528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394527    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bhd\" (UniqueName: \"kubernetes.io/projected/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-kube-api-access-n6bhd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394546    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-sys-fs\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394571    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw6jc\" (UniqueName: \"kubernetes.io/projected/69042048-b2e6-477d-a330-94d2a1e608e3-kube-api-access-nw6jc\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394594    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/378ce7f3-c33f-44f5-a383-a8acc714f547-env-overrides\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394596    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-ovn\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394589    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58899f4-efb6-4d21-b1eb-fcea8c046356-data-volume\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394748    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-node-log\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394786    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-var-lib-kubelet\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394793    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2e2e850-03eb-485d-940e-1b6d9063091c-iptables-alerter-script\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394834    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-host\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.394941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394906    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba1ed75b-f38b-4589-87fa-0c8d22abe784-serviceca\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394954    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-log-socket\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.394980    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395006    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-device-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395029    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-conf\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395059    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/375dc3d6-4c9f-4f39-96b8-a2dec49d37f6-signing-cabundle\") pod \"service-ca-5fffc97bdf-vmxp7\" (UID: \"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395096    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-multus-certs\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395121    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba1ed75b-f38b-4589-87fa-0c8d22abe784-serviceca\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395143    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg67\" (UniqueName: \"kubernetes.io/projected/b6257887-6b6e-4ad3-ae76-3d121659e2d4-kube-api-access-shg67\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395169    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfvj\" (UniqueName: \"kubernetes.io/projected/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-kube-api-access-llfvj\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395195    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395219    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-lib-modules\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.395254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395243    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-tuned\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395265    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-tmp\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395289    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-systemd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395356    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395398    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-tuned\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395406    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-tmp\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395411    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-socket-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395452    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395477    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/375dc3d6-4c9f-4f39-96b8-a2dec49d37f6-signing-key\") pod \"service-ca-5fffc97bdf-vmxp7\" (UID: \"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395517    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6lw\" (UniqueName: \"kubernetes.io/projected/375dc3d6-4c9f-4f39-96b8-a2dec49d37f6-kube-api-access-2n6lw\") pod \"service-ca-5fffc97bdf-vmxp7\" (UID: \"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395521    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395543    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395568    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cnibin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395599    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cni-binary-copy\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395608    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395630    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-conf-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395691    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395737    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395738    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg2kd\" (UniqueName: \"kubernetes.io/projected/4da03b13-3a30-48de-b399-ba82b071c456-kube-api-access-mg2kd\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.395763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395769    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cni-binary-copy\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395774    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-kubelet\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395861    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-netd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395886    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/378ce7f3-c33f-44f5-a383-a8acc714f547-ovn-node-metrics-cert\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395900    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395949    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-multus\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.395977    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-cert\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396050    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/378ce7f3-c33f-44f5-a383-a8acc714f547-ovn-node-metrics-cert\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396076    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396112    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-var-lib-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396133    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-cert\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396254    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-bin\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396276    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4100403-c8f8-4ac1-98c4-2795d5a11b10-tmp-dir\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396292    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-bin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396304    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4da03b13-3a30-48de-b399-ba82b071c456-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396308    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396334    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396371    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjw8\" (UniqueName: \"kubernetes.io/projected/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-api-access-cpjw8\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396388    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396395    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4100403-c8f8-4ac1-98c4-2795d5a11b10-tmp-dir\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396405    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-dbus\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396420    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqb6m\" (UniqueName: \"kubernetes.io/projected/d237eda3-f867-4dd3-8a9b-24aaa9c37403-kube-api-access-tqb6m\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:33.396503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396512    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-netns\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.397155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396558    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4100403-c8f8-4ac1-98c4-2795d5a11b10-config-volume\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:33.397155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396616    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vflt4\" (UniqueName: \"kubernetes.io/projected/1b6197ff-1085-4ade-a095-688ff39c5f9c-kube-api-access-vflt4\") pod \"migrator-66df5d496b-6b6r5\" (UID: \"1b6197ff-1085-4ade-a095-688ff39c5f9c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"
Nov 03 13:08:33.397155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396678    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.397155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.396728    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4100403-c8f8-4ac1-98c4-2795d5a11b10-config-volume\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:33.398722 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.398702    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czgz\" (UniqueName: \"kubernetes.io/projected/b2e2e850-03eb-485d-940e-1b6d9063091c-kube-api-access-4czgz\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:08:33.398939 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.398925    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrblj\" (UniqueName: \"kubernetes.io/projected/b4100403-c8f8-4ac1-98c4-2795d5a11b10-kube-api-access-vrblj\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:33.398978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.398963    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t69vq\" (UniqueName: \"kubernetes.io/projected/378ce7f3-c33f-44f5-a383-a8acc714f547-kube-api-access-t69vq\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.401661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.401642    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmwl\" (UniqueName: \"kubernetes.io/projected/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-kube-api-access-lrmwl\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:08:33.401826 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.401811    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9k4v\" (UniqueName: \"kubernetes.io/projected/ba1ed75b-f38b-4589-87fa-0c8d22abe784-kube-api-access-l9k4v\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:08:33.404107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.404088    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw6jc\" (UniqueName: \"kubernetes.io/projected/69042048-b2e6-477d-a330-94d2a1e608e3-kube-api-access-nw6jc\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.404376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.404359    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqb6m\" (UniqueName: \"kubernetes.io/projected/d237eda3-f867-4dd3-8a9b-24aaa9c37403-kube-api-access-tqb6m\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:33.404478 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.404454    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vflt4\" (UniqueName: \"kubernetes.io/projected/1b6197ff-1085-4ade-a095-688ff39c5f9c-kube-api-access-vflt4\") pod \"migrator-66df5d496b-6b6r5\" (UID: \"1b6197ff-1085-4ade-a095-688ff39c5f9c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5"
Nov 03 13:08:33.404568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.404547    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bhd\" (UniqueName: \"kubernetes.io/projected/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-kube-api-access-n6bhd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.404738 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.404709    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg67\" (UniqueName: \"kubernetes.io/projected/b6257887-6b6e-4ad3-ae76-3d121659e2d4-kube-api-access-shg67\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.404799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.404779    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xwvb\" (UniqueName: \"kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb\") pod \"periodic-gathering-f9xld-h6jgs\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") " pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:08:33.404938 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.404922    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjw8\" (UniqueName: \"kubernetes.io/projected/d58899f4-efb6-4d21-b1eb-fcea8c046356-kube-api-access-cpjw8\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.405928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.405908    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rrw\" (UniqueName: \"kubernetes.io/projected/70f3bdac-e826-4187-b83c-993fe2c5d7ac-kube-api-access-w9rrw\") pod \"network-check-target-ch6h8\" (UID: \"70f3bdac-e826-4187-b83c-993fe2c5d7ac\") " pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:33.406949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.406928    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg2kd\" (UniqueName: \"kubernetes.io/projected/4da03b13-3a30-48de-b399-ba82b071c456-kube-api-access-mg2kd\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.407107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.407092    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfvj\" (UniqueName: \"kubernetes.io/projected/e0bc4a29-7ff7-4b22-8f9f-56906b70ee41-kube-api-access-llfvj\") pod \"ingress-canary-x7hff\" (UID: \"e0bc4a29-7ff7-4b22-8f9f-56906b70ee41\") " pod="openshift-ingress-canary/ingress-canary-x7hff"
Nov 03 13:08:33.417197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.417174    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 13:08:33.417247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.417180    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k8z6b\""
Nov 03 13:08:33.418871 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.418858    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 13:08:33.418949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.418933    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 13:08:33.418980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.418953    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 13:08:33.497143 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497108    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-kubelet-config\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:33.497143 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497141    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497160    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-systemd-units\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497203    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-systemd-units\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497201    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-slash\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497234    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-etc-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497239    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-slash\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497244    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497249    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-system-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497273    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-etc-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497293    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-system-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497299    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497317    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-k8s-cni-cncf-io\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497324    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-kubelet-config\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:33.497358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497333    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-netns\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497400    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-modprobe-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497408    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-k8s-cni-cncf-io\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497365    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-netns\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497429    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysconfig\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497452    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:33.497458    4447 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497481    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-kubelet\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497508    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d58899f4-efb6-4d21-b1eb-fcea8c046356-crio-socket\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497540    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-modprobe-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:33.497547    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:33.997527056 +0000 UTC m=+1.809195240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : secret "metrics-daemon-secret" not found
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497565    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-system-cni-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497572    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d58899f4-efb6-4d21-b1eb-fcea8c046356-crio-socket\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497575    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-kubelet\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497541    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysconfig\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497583    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-cnibin\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497618    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-d\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497640    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-cnibin\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497678    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-hostroot\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497693    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4da03b13-3a30-48de-b399-ba82b071c456-system-cni-dir\") pod \"multus-additional-cni-plugins-gjtf7\" (UID: \"4da03b13-3a30-48de-b399-ba82b071c456\") " pod="openshift-multus/multus-additional-cni-plugins-gjtf7"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497704    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-etc-kubernetes\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497719    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-hostroot\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497731    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497756    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-etc-kubernetes\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497757    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-systemd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497791    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-run\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:33.497789    4447 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497804    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-sys\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497823    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-registration-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:33.497836    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:33.99782517 +0000 UTC m=+1.809493336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497841    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-run\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497804    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-systemd\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497864    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-kubernetes\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497874    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-registration-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497882    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-sys\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497888    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba1ed75b-f38b-4589-87fa-0c8d22abe784-host\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497908    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-kubernetes\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497918    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba1ed75b-f38b-4589-87fa-0c8d22abe784-host\") pod \"node-ca-88kk5\" (UID: \"ba1ed75b-f38b-4589-87fa-0c8d22abe784\") " pod="openshift-image-registry/node-ca-88kk5"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497913    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497934    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497952    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-sys-fs\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497968    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-ovn\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.497953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497979    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-sys-fs\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.497983    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-node-log\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498008    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-var-lib-kubelet\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498018    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-ovn\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498029    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-host\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498054    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-node-log\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498055    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-log-socket\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498089    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-log-socket\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498092    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-var-lib-kubelet\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498086    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498118    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498127    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-device-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498097    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-host\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498144    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-conf\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498173    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-device-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498187    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/375dc3d6-4c9f-4f39-96b8-a2dec49d37f6-signing-cabundle\") pod \"service-ca-5fffc97bdf-vmxp7\" (UID: \"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498208    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-multus-certs\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498232    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-lib-modules\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498258    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-systemd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498263    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-etc-sysctl-conf\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498274    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-run-multus-certs\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498294    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498302    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-systemd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498338    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-run-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498359    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-socket-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498387    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/375dc3d6-4c9f-4f39-96b8-a2dec49d37f6-signing-key\") pod \"service-ca-5fffc97bdf-vmxp7\" (UID: \"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498405    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-socket-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498411    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6lw\" (UniqueName: \"kubernetes.io/projected/375dc3d6-4c9f-4f39-96b8-a2dec49d37f6-kube-api-access-2n6lw\") pod \"service-ca-5fffc97bdf-vmxp7\" (UID: \"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498436    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cnibin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498361    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12-lib-modules\") pod \"tuned-btbj2\" (UID: \"586f11e1-e0bd-439c-9aa3-2cb5eb3d2e12\") " pod="openshift-cluster-node-tuning-operator/tuned-btbj2"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498487    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-cnibin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498519    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-conf-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498545    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498554    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-conf-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498570    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-kubelet\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498592    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-netd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498605    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498616    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498618    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-kubelet\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498660    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-netd\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:33.498675    4447 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498713    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-multus\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:33.498721    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls podName:b4100403-c8f8-4ac1-98c4-2795d5a11b10 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:33.998710904 +0000 UTC m=+1.810379070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls") pod "dns-default-v5xss" (UID: "b4100403-c8f8-4ac1-98c4-2795d5a11b10") : secret "dns-default-metrics-tls" not found
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498750    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-multus\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498765    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-var-lib-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498797    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-bin\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498766    4447 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498824    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-bin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498842    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-cni-bin\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498847    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498873    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-dbus\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498879    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/69042048-b2e6-477d-a330-94d2a1e608e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnqvf\" (UID: \"69042048-b2e6-477d-a330-94d2a1e608e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnqvf"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498883    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-host-var-lib-cni-bin\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498802    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-var-lib-openvswitch\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498899    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-netns\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498932    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498936    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/375dc3d6-4c9f-4f39-96b8-a2dec49d37f6-signing-cabundle\") pod \"service-ca-5fffc97bdf-vmxp7\" (UID: \"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498937    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/378ce7f3-c33f-44f5-a383-a8acc714f547-host-run-netns\") pod \"ovnkube-node-62dpn\" (UID: \"378ce7f3-c33f-44f5-a383-a8acc714f547\") " pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498941    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ef0c534-57ad-4269-8d12-e2a5879a3eed-dbus\") pod \"global-pull-secret-syncer-9wrnl\" (UID: \"0ef0c534-57ad-4269-8d12-e2a5879a3eed\") " pod="kube-system/global-pull-secret-syncer-9wrnl"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498953    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-socket-dir-parent\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498984    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-cni-dir\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498990    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6257887-6b6e-4ad3-ae76-3d121659e2d4-multus-socket-dir-parent\") pod \"multus-mkzmk\" (UID: \"b6257887-6b6e-4ad3-ae76-3d121659e2d4\") " pod="openshift-multus/multus-mkzmk"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.498994    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-hosts-file\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.499018    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2e2e850-03eb-485d-940e-1b6d9063091c-host-slash\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.499048    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44f5fcad-a6fb-4bd2-99aa-f4973415aafa-hosts-file\") pod \"node-resolver-fgqkh\" (UID: \"44f5fcad-a6fb-4bd2-99aa-f4973415aafa\") " pod="openshift-dns/node-resolver-fgqkh"
Nov 03 13:08:33.499284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.499070    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2e2e850-03eb-485d-940e-1b6d9063091c-host-slash\") pod \"iptables-alerter-2fwt9\" (UID: \"b2e2e850-03eb-485d-940e-1b6d9063091c\") " pod="openshift-network-operator/iptables-alerter-2fwt9"
Nov 03 13:08:33.501119 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.500983    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/375dc3d6-4c9f-4f39-96b8-a2dec49d37f6-signing-key\") pod \"service-ca-5fffc97bdf-vmxp7\" (UID: \"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.506995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.506973    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6lw\" (UniqueName: \"kubernetes.io/projected/375dc3d6-4c9f-4f39-96b8-a2dec49d37f6-kube-api-access-2n6lw\") pod \"service-ca-5fffc97bdf-vmxp7\" (UID: \"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.620430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.620408    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:08:33.669823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.669792    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7"
Nov 03 13:08:33.953475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:33.953442    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vmxp7"]
Nov 03 13:08:33.957433 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:33.957398    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375dc3d6_4c9f_4f39_96b8_a2dec49d37f6.slice/crio-737013a5303bda6b4dc2a2a6c3917bc27767e6ceb37b9261962ef8c709e82281 WatchSource:0}: Error finding container 737013a5303bda6b4dc2a2a6c3917bc27767e6ceb37b9261962ef8c709e82281: Status 404 returned error can't find the container with id 737013a5303bda6b4dc2a2a6c3917bc27767e6ceb37b9261962ef8c709e82281
Nov 03 13:08:34.003634 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:34.003609    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:34.003768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:34.003650    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:34.003768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:34.003674    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:34.003768 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:34.003762    4447 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:34.003905 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:34.003763    4447 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 13:08:34.003905 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:34.003766    4447 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 13:08:34.003905 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:34.003803    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:35.00379187 +0000 UTC m=+2.815460049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:34.003905 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:34.003837    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:35.00382328 +0000 UTC m=+2.815491447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : secret "metrics-daemon-secret" not found
Nov 03 13:08:34.003905 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:34.003848    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls podName:b4100403-c8f8-4ac1-98c4-2795d5a11b10 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:35.003842467 +0000 UTC m=+2.815510633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls") pod "dns-default-v5xss" (UID: "b4100403-c8f8-4ac1-98c4-2795d5a11b10") : secret "dns-default-metrics-tls" not found
Nov 03 13:08:34.353004 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:34.352915    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7" event={"ID":"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6","Type":"ContainerStarted","Data":"737013a5303bda6b4dc2a2a6c3917bc27767e6ceb37b9261962ef8c709e82281"}
Nov 03 13:08:35.011871 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:35.011822    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:35.012040 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:35.011904    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:35.012040 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:35.011969    4447 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 13:08:35.012040 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:35.011982    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:35.012040 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:35.012034    4447 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:35.012220 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:35.012095    4447 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 13:08:35.012220 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:35.012102    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:37.012081607 +0000 UTC m=+4.823749789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : secret "metrics-daemon-secret" not found
Nov 03 13:08:35.012220 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:35.012175    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:37.012153475 +0000 UTC m=+4.823821648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:35.012220 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:35.012195    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls podName:b4100403-c8f8-4ac1-98c4-2795d5a11b10 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:37.012184831 +0000 UTC m=+4.823853003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls") pod "dns-default-v5xss" (UID: "b4100403-c8f8-4ac1-98c4-2795d5a11b10") : secret "dns-default-metrics-tls" not found
Nov 03 13:08:35.356514 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:35.356484    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6b6r5" event={"ID":"1b6197ff-1085-4ade-a095-688ff39c5f9c","Type":"ContainerStarted","Data":"6afb8d1e6ee552d389ae20fe2dc56bcfd7f891333a03d55fad6df3f2ce34f6f8"}
Nov 03 13:08:36.360179 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:36.360102    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7" event={"ID":"375dc3d6-4c9f-4f39-96b8-a2dec49d37f6","Type":"ContainerStarted","Data":"fee96535e2753db2cc9a6fc9f601da955fe81fc90e2738bf604087e8f6e42f35"}
Nov 03 13:08:36.378855 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:36.378785    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-vmxp7" podStartSLOduration=2.397502955 podStartE2EDuration="4.378771009s" totalImagesPullingTime="1.981268054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:36.378484005 +0000 UTC m=+4.190152209" watchObservedRunningTime="2026-11-03 13:08:36.378771009 +0000 UTC m=+4.190439197"
Nov 03 13:08:36.760658 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:36.760611    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:08:37.027812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:37.027724    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:37.027812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:37.027780    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:37.028005 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:37.027817    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:37.028005 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:37.027890    4447 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 13:08:37.028005 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:37.027908    4447 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:37.028005 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:37.027895    4447 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 13:08:37.028005 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:37.027961    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls podName:d58899f4-efb6-4d21-b1eb-fcea8c046356 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:41.027945347 +0000 UTC m=+8.839613532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqvjq" (UID: "d58899f4-efb6-4d21-b1eb-fcea8c046356") : secret "insights-runtime-extractor-tls" not found
Nov 03 13:08:37.028005 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:37.027975    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:41.027969436 +0000 UTC m=+8.839637602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : secret "metrics-daemon-secret" not found
Nov 03 13:08:37.028005 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:37.027984    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls podName:b4100403-c8f8-4ac1-98c4-2795d5a11b10 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:41.027979554 +0000 UTC m=+8.839647719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls") pod "dns-default-v5xss" (UID: "b4100403-c8f8-4ac1-98c4-2795d5a11b10") : secret "dns-default-metrics-tls" not found
Nov 03 13:08:39.830605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:39.830575    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:40.189634 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:40.189608    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:40.369743 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:40.369711    4447 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 13:08:41.057532 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.057495    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:41.057532 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.057544    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:41.057988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.057570    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:41.057988 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:41.057671    4447 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 13:08:41.057988 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:08:41.057751    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs podName:d237eda3-f867-4dd3-8a9b-24aaa9c37403 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:49.057733861 +0000 UTC m=+16.869402040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs") pod "network-metrics-daemon-2m9tv" (UID: "d237eda3-f867-4dd3-8a9b-24aaa9c37403") : secret "metrics-daemon-secret" not found
Nov 03 13:08:41.059961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.059938    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d58899f4-efb6-4d21-b1eb-fcea8c046356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqvjq\" (UID: \"d58899f4-efb6-4d21-b1eb-fcea8c046356\") " pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:41.060066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.059999    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4100403-c8f8-4ac1-98c4-2795d5a11b10-metrics-tls\") pod \"dns-default-v5xss\" (UID: \"b4100403-c8f8-4ac1-98c4-2795d5a11b10\") " pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:41.117103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.117066    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:41.119166 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.119141    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqvjq"
Nov 03 13:08:41.215324 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.215295    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-62dpn"
Nov 03 13:08:41.394724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.394640    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v5xss"]
Nov 03 13:08:41.396930 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.396902    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dqvjq"]
Nov 03 13:08:41.398364 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:41.398329    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4100403_c8f8_4ac1_98c4_2795d5a11b10.slice/crio-fc9b1f227cf9af09a278c84da50978c2124f72950bf54cbf486d8e52c7f4d07f WatchSource:0}: Error finding container fc9b1f227cf9af09a278c84da50978c2124f72950bf54cbf486d8e52c7f4d07f: Status 404 returned error can't find the container with id fc9b1f227cf9af09a278c84da50978c2124f72950bf54cbf486d8e52c7f4d07f
Nov 03 13:08:41.401162 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:41.401140    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58899f4_efb6_4d21_b1eb_fcea8c046356.slice/crio-aac53a3fbfed1acaebc8f622897e7f7393d549a0d182b75900e550c1ace0b482 WatchSource:0}: Error finding container aac53a3fbfed1acaebc8f622897e7f7393d549a0d182b75900e550c1ace0b482: Status 404 returned error can't find the container with id aac53a3fbfed1acaebc8f622897e7f7393d549a0d182b75900e550c1ace0b482
Nov 03 13:08:41.456302 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:41.456284    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d9722"
Nov 03 13:08:42.377582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:42.377551    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqvjq" event={"ID":"d58899f4-efb6-4d21-b1eb-fcea8c046356","Type":"ContainerStarted","Data":"d2c18a3152195f00e5f1cb46fc1b460253aa58f416638697d34d874d675ee3cc"}
Nov 03 13:08:42.378034 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:42.377614    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqvjq" event={"ID":"d58899f4-efb6-4d21-b1eb-fcea8c046356","Type":"ContainerStarted","Data":"aac53a3fbfed1acaebc8f622897e7f7393d549a0d182b75900e550c1ace0b482"}
Nov 03 13:08:42.378916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:42.378882    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v5xss" event={"ID":"b4100403-c8f8-4ac1-98c4-2795d5a11b10","Type":"ContainerStarted","Data":"fc9b1f227cf9af09a278c84da50978c2124f72950bf54cbf486d8e52c7f4d07f"}
Nov 03 13:08:43.384058 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:43.383972    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqvjq" event={"ID":"d58899f4-efb6-4d21-b1eb-fcea8c046356","Type":"ContainerStarted","Data":"3d9789f1f126b05936a699a9bb834a2eedd1eac55fb4ea3bdf345e77186e3bbe"}
Nov 03 13:08:43.385999 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:43.385972    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v5xss" event={"ID":"b4100403-c8f8-4ac1-98c4-2795d5a11b10","Type":"ContainerStarted","Data":"e8f534567cf21557e83db63713685e106b4f4bb5a17c8d97d55c05093b47455f"}
Nov 03 13:08:43.386104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:43.386023    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v5xss" event={"ID":"b4100403-c8f8-4ac1-98c4-2795d5a11b10","Type":"ContainerStarted","Data":"3c7e693e85e726331e325929f748359c82a7301a1bd47e0b1b5d9b6aff512181"}
Nov 03 13:08:43.386156 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:43.386120    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:44.391366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:44.391306    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqvjq" event={"ID":"d58899f4-efb6-4d21-b1eb-fcea8c046356","Type":"ContainerStarted","Data":"74df17ef9ceb5596fd728628edb7cb3e0d06a1e3cea5b361fe1788dc3e6acc83"}
Nov 03 13:08:49.118227 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:49.118191    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:49.120473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:49.120448    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d237eda3-f867-4dd3-8a9b-24aaa9c37403-metrics-certs\") pod \"network-metrics-daemon-2m9tv\" (UID: \"d237eda3-f867-4dd3-8a9b-24aaa9c37403\") " pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:49.214539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:49.214491    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2m9tv"
Nov 03 13:08:49.483070 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:49.483035    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2m9tv"]
Nov 03 13:08:49.486815 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:49.486784    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd237eda3_f867_4dd3_8a9b_24aaa9c37403.slice/crio-db14d579a70ed196dd504e153536f7a7dd3195470f06e4e57a513b4bbb97a5e7 WatchSource:0}: Error finding container db14d579a70ed196dd504e153536f7a7dd3195470f06e4e57a513b4bbb97a5e7: Status 404 returned error can't find the container with id db14d579a70ed196dd504e153536f7a7dd3195470f06e4e57a513b4bbb97a5e7
Nov 03 13:08:50.410290 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:50.410254    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2m9tv" event={"ID":"d237eda3-f867-4dd3-8a9b-24aaa9c37403","Type":"ContainerStarted","Data":"db14d579a70ed196dd504e153536f7a7dd3195470f06e4e57a513b4bbb97a5e7"}
Nov 03 13:08:51.415476 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:51.415443    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2m9tv" event={"ID":"d237eda3-f867-4dd3-8a9b-24aaa9c37403","Type":"ContainerStarted","Data":"6a109acac99da5b6f1ab39342e4c413b080ff6b52640ed15edd9e68fb649cb7d"}
Nov 03 13:08:51.415845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:51.415508    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2m9tv" event={"ID":"d237eda3-f867-4dd3-8a9b-24aaa9c37403","Type":"ContainerStarted","Data":"896a3f926289537adf69c849ea337f496d5e1086b13009e5d5dbaefcf5737ffc"}
Nov 03 13:08:53.336922 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.336802    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-p694g-sqtcz"]
Nov 03 13:08:53.338809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.338788    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.362845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.362817    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p694g-sqtcz"]
Nov 03 13:08:53.376329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.376306    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-j565n"]
Nov 03 13:08:53.378283 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.378262    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:08:53.392562 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.392542    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 13:08:53.393624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.393606    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v5xss"
Nov 03 13:08:53.399104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.399090    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-vvvm2\""
Nov 03 13:08:53.399171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.399129    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 13:08:53.402437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.402411    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j565n"]
Nov 03 13:08:53.430196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.430155    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cc78ccb-6628n"]
Nov 03 13:08:53.432059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.432037    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.448462 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.448439    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da37367-7ecb-4213-a856-c7ae36609bae-tmp\") pod \"downloads-6dd789c8b7-j565n\" (UID: \"6da37367-7ecb-4213-a856-c7ae36609bae\") " pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:08:53.448577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.448466    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ef0a05-3328-401e-946c-cc7b1582b15c-archives-path\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.448577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.448482    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58gv\" (UniqueName: \"kubernetes.io/projected/f5ef0a05-3328-401e-946c-cc7b1582b15c-kube-api-access-m58gv\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.448577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.448540    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-service-ca-bundle\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.448717 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.448581    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdsf8\" (UniqueName: \"kubernetes.io/projected/6da37367-7ecb-4213-a856-c7ae36609bae-kube-api-access-hdsf8\") pod \"downloads-6dd789c8b7-j565n\" (UID: \"6da37367-7ecb-4213-a856-c7ae36609bae\") " pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:08:53.448717 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.448621    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-trusted-ca-bundle\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.468602 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.468580    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 13:08:53.468849 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.468831    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 13:08:53.468849 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.468841    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 13:08:53.470835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.470801    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc78ccb-6628n"]
Nov 03 13:08:53.498527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.498502    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 13:08:53.498662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.498621    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 13:08:53.498662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.498625    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z5dpr\""
Nov 03 13:08:53.512612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.512586    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"]
Nov 03 13:08:53.514637 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.514622    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"
Nov 03 13:08:53.526916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.526894    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"]
Nov 03 13:08:53.549465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549442    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da37367-7ecb-4213-a856-c7ae36609bae-tmp\") pod \"downloads-6dd789c8b7-j565n\" (UID: \"6da37367-7ecb-4213-a856-c7ae36609bae\") " pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:08:53.549583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549476    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-oauth-serving-cert\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.549583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549503    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58gv\" (UniqueName: \"kubernetes.io/projected/f5ef0a05-3328-401e-946c-cc7b1582b15c-kube-api-access-m58gv\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.549583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549531    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdsf8\" (UniqueName: \"kubernetes.io/projected/6da37367-7ecb-4213-a856-c7ae36609bae-kube-api-access-hdsf8\") pod \"downloads-6dd789c8b7-j565n\" (UID: \"6da37367-7ecb-4213-a856-c7ae36609bae\") " pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:08:53.549801 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549635    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlmrw\" (UniqueName: \"kubernetes.io/projected/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-kube-api-access-nlmrw\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.549801 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549674    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-tmp\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.549801 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549788    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-service-ca-bundle\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.550522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549844    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-trusted-ca-bundle\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.550522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549889    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da37367-7ecb-4213-a856-c7ae36609bae-tmp\") pod \"downloads-6dd789c8b7-j565n\" (UID: \"6da37367-7ecb-4213-a856-c7ae36609bae\") " pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:08:53.550522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549897    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-service-ca\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.550522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.549920    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ef0a05-3328-401e-946c-cc7b1582b15c-archives-path\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.550522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.550122    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-serving-cert\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.550522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.550194    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-oauth-config\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.550522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.550263    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-config\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.550522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.550363    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ef0a05-3328-401e-946c-cc7b1582b15c-archives-path\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.550792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.550668    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-service-ca-bundle\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.550853 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.550837    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-trusted-ca-bundle\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.575640 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.575613    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58gv\" (UniqueName: \"kubernetes.io/projected/f5ef0a05-3328-401e-946c-cc7b1582b15c-kube-api-access-m58gv\") pod \"periodic-gathering-p694g-sqtcz\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") " pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.581165 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.581146    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dcf6c45d8-f5t74"]
Nov 03 13:08:53.582948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.582931    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdsf8\" (UniqueName: \"kubernetes.io/projected/6da37367-7ecb-4213-a856-c7ae36609bae-kube-api-access-hdsf8\") pod \"downloads-6dd789c8b7-j565n\" (UID: \"6da37367-7ecb-4213-a856-c7ae36609bae\") " pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:08:53.583433 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.583420    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.593540 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.593481    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 13:08:53.593643 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.593568    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qn58d\""
Nov 03 13:08:53.598567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.598550    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 13:08:53.648034 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.648003    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:08:53.650434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650411    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dcf6c45d8-f5t74"]
Nov 03 13:08:53.650753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650729    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-serving-cert\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.650849 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650770    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-oauth-config\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.650849 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650797    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-config\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.650849 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650828    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6257b998-d021-47de-850b-65d0c5e03eb0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lp9rq\" (UID: \"6257b998-d021-47de-850b-65d0c5e03eb0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"
Nov 03 13:08:53.650959 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650859    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-oauth-serving-cert\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.650959 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650883    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39f044b1-cfab-41fb-a72a-819e22c431cf-installation-pull-secrets\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.650959 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650917    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39f044b1-cfab-41fb-a72a-819e22c431cf-registry-tls\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.650959 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650938    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39f044b1-cfab-41fb-a72a-819e22c431cf-bound-sa-token\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.651087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650967    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmrw\" (UniqueName: \"kubernetes.io/projected/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-kube-api-access-nlmrw\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.651087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.650990    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzwr\" (UniqueName: \"kubernetes.io/projected/39f044b1-cfab-41fb-a72a-819e22c431cf-kube-api-access-wjzwr\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.651087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.651018    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-tmp\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.651087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.651040    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39f044b1-cfab-41fb-a72a-819e22c431cf-ca-trust-extracted\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.651087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.651070    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39f044b1-cfab-41fb-a72a-819e22c431cf-image-registry-private-configuration\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.651308 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.651106    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39f044b1-cfab-41fb-a72a-819e22c431cf-registry-certificates\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.651808 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.651786    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-tmp\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.652568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.652543    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-config\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.652664 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.652593    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39f044b1-cfab-41fb-a72a-819e22c431cf-trusted-ca\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.652664 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.652621    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-service-ca\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.652836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.652805    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-oauth-serving-cert\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.653270 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.653250    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-service-ca\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.653868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.653847    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-oauth-config\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.654410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.654393    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-serving-cert\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.664208 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.664185    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlmrw\" (UniqueName: \"kubernetes.io/projected/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-kube-api-access-nlmrw\") pod \"console-cc78ccb-6628n\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") " pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.687886 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.687841    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:08:53.696557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.695793    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 13:08:53.696557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.695936    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bbkgd\""
Nov 03 13:08:53.696557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.696029    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 13:08:53.696557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.696091    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 13:08:53.742697 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.742673    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:08:53.754028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.753486    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39f044b1-cfab-41fb-a72a-819e22c431cf-trusted-ca\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.754028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.753542    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6257b998-d021-47de-850b-65d0c5e03eb0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lp9rq\" (UID: \"6257b998-d021-47de-850b-65d0c5e03eb0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"
Nov 03 13:08:53.754028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.753577    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39f044b1-cfab-41fb-a72a-819e22c431cf-installation-pull-secrets\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.754247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.754067    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39f044b1-cfab-41fb-a72a-819e22c431cf-registry-tls\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.754247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.754107    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39f044b1-cfab-41fb-a72a-819e22c431cf-bound-sa-token\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.754247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.754143    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzwr\" (UniqueName: \"kubernetes.io/projected/39f044b1-cfab-41fb-a72a-819e22c431cf-kube-api-access-wjzwr\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.754247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.754172    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39f044b1-cfab-41fb-a72a-819e22c431cf-ca-trust-extracted\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.754247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.754207    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39f044b1-cfab-41fb-a72a-819e22c431cf-image-registry-private-configuration\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.754247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.754238    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39f044b1-cfab-41fb-a72a-819e22c431cf-registry-certificates\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.754622 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.754587    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39f044b1-cfab-41fb-a72a-819e22c431cf-ca-trust-extracted\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.754679 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.754662    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39f044b1-cfab-41fb-a72a-819e22c431cf-trusted-ca\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.755184 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.755161    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39f044b1-cfab-41fb-a72a-819e22c431cf-registry-certificates\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.757411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.757387    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39f044b1-cfab-41fb-a72a-819e22c431cf-image-registry-private-configuration\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.757752 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.757723    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39f044b1-cfab-41fb-a72a-819e22c431cf-installation-pull-secrets\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.758120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.758102    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39f044b1-cfab-41fb-a72a-819e22c431cf-registry-tls\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.758973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.758953    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6257b998-d021-47de-850b-65d0c5e03eb0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lp9rq\" (UID: \"6257b998-d021-47de-850b-65d0c5e03eb0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"
Nov 03 13:08:53.766073 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.766050    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39f044b1-cfab-41fb-a72a-819e22c431cf-bound-sa-token\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.766557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.766539    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzwr\" (UniqueName: \"kubernetes.io/projected/39f044b1-cfab-41fb-a72a-819e22c431cf-kube-api-access-wjzwr\") pod \"image-registry-5dcf6c45d8-f5t74\" (UID: \"39f044b1-cfab-41fb-a72a-819e22c431cf\") " pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.826986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.826948    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"
Nov 03 13:08:53.898759 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.898729    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:53.916370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.916326    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p694g-sqtcz"]
Nov 03 13:08:53.919300 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:53.919273    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ef0a05_3328_401e_946c_cc7b1582b15c.slice/crio-721763a0989d01cc4d28fe714acd6b646a63ae21a75f83b4f62d8630bacdf8dc WatchSource:0}: Error finding container 721763a0989d01cc4d28fe714acd6b646a63ae21a75f83b4f62d8630bacdf8dc: Status 404 returned error can't find the container with id 721763a0989d01cc4d28fe714acd6b646a63ae21a75f83b4f62d8630bacdf8dc
Nov 03 13:08:53.968105 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:53.968074    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j565n"]
Nov 03 13:08:53.974268 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:53.974244    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da37367_7ecb_4213_a856_c7ae36609bae.slice/crio-8aabcd898a47186f27928bdf43161b2f39454591d7ee71894331b2320fa2ef2f WatchSource:0}: Error finding container 8aabcd898a47186f27928bdf43161b2f39454591d7ee71894331b2320fa2ef2f: Status 404 returned error can't find the container with id 8aabcd898a47186f27928bdf43161b2f39454591d7ee71894331b2320fa2ef2f
Nov 03 13:08:54.018693 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.018639    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc78ccb-6628n"]
Nov 03 13:08:54.022084 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:54.022057    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b80df0_a7c0_44e7_b7ee_31a565e1799b.slice/crio-424fdf925caf6b07c7b302dd47688980f0763c50da21e7f2abefd71c30d24637 WatchSource:0}: Error finding container 424fdf925caf6b07c7b302dd47688980f0763c50da21e7f2abefd71c30d24637: Status 404 returned error can't find the container with id 424fdf925caf6b07c7b302dd47688980f0763c50da21e7f2abefd71c30d24637
Nov 03 13:08:54.094584 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.094552    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"]
Nov 03 13:08:54.168525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.168450    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dcf6c45d8-f5t74"]
Nov 03 13:08:54.171977 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:54.171953    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f044b1_cfab_41fb_a72a_819e22c431cf.slice/crio-c916d79fc431a0a7399b66f29501a4f1626288abd7df4845078e0bbf5e93f0db WatchSource:0}: Error finding container c916d79fc431a0a7399b66f29501a4f1626288abd7df4845078e0bbf5e93f0db: Status 404 returned error can't find the container with id c916d79fc431a0a7399b66f29501a4f1626288abd7df4845078e0bbf5e93f0db
Nov 03 13:08:54.430331 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.430296    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc78ccb-6628n" event={"ID":"a0b80df0-a7c0-44e7-b7ee-31a565e1799b","Type":"ContainerStarted","Data":"424fdf925caf6b07c7b302dd47688980f0763c50da21e7f2abefd71c30d24637"}
Nov 03 13:08:54.431915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.431888    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j565n" event={"ID":"6da37367-7ecb-4213-a856-c7ae36609bae","Type":"ContainerStarted","Data":"8aabcd898a47186f27928bdf43161b2f39454591d7ee71894331b2320fa2ef2f"}
Nov 03 13:08:54.433894 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.433864    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74" event={"ID":"39f044b1-cfab-41fb-a72a-819e22c431cf","Type":"ContainerStarted","Data":"359c0a6e2c44817326c1d419589c015a0ea3c579973e30648ee978f7071a71f2"}
Nov 03 13:08:54.434013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.433916    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74" event={"ID":"39f044b1-cfab-41fb-a72a-819e22c431cf","Type":"ContainerStarted","Data":"c916d79fc431a0a7399b66f29501a4f1626288abd7df4845078e0bbf5e93f0db"}
Nov 03 13:08:54.434013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.433945    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:08:54.435491 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.435464    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq" event={"ID":"6257b998-d021-47de-850b-65d0c5e03eb0","Type":"ContainerStarted","Data":"a2e1091a14bf047f939238d045041490f3081b79031f35f244e8cf74063d0411"}
Nov 03 13:08:54.437165 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.437136    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p694g-sqtcz" event={"ID":"f5ef0a05-3328-401e-946c-cc7b1582b15c","Type":"ContainerStarted","Data":"83686f3d151d98f57bd8c7cbc3d43815daa57a2a41e51817457a725cbc5a0e6f"}
Nov 03 13:08:54.437253 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.437204    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p694g-sqtcz" event={"ID":"f5ef0a05-3328-401e-946c-cc7b1582b15c","Type":"ContainerStarted","Data":"721763a0989d01cc4d28fe714acd6b646a63ae21a75f83b4f62d8630bacdf8dc"}
Nov 03 13:08:54.454809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.454762    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74" podStartSLOduration=1.454748065 podStartE2EDuration="1.454748065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:54.454279487 +0000 UTC m=+22.265947706" watchObservedRunningTime="2026-11-03 13:08:54.454748065 +0000 UTC m=+22.266416256"
Nov 03 13:08:54.468878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.468187    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-p694g-sqtcz" podStartSLOduration=1.468170678 podStartE2EDuration="1.468170678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:54.467929211 +0000 UTC m=+22.279597418" watchObservedRunningTime="2026-11-03 13:08:54.468170678 +0000 UTC m=+22.279838866"
Nov 03 13:08:54.875666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:54.875594    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v5xss_b4100403-c8f8-4ac1-98c4-2795d5a11b10/dns/0.log"
Nov 03 13:08:55.073539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:55.073509    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v5xss_b4100403-c8f8-4ac1-98c4-2795d5a11b10/kube-rbac-proxy/0.log"
Nov 03 13:08:55.445086 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:55.445052    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq" event={"ID":"6257b998-d021-47de-850b-65d0c5e03eb0","Type":"ContainerStarted","Data":"6ff0ef96854ac6b58dd8d0e5a6aeadcfb4bdf9c891c3f3489d54213caad589bb"}
Nov 03 13:08:55.445557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:55.445519    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"
Nov 03 13:08:55.451441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:55.451415    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq"
Nov 03 13:08:55.459704 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:55.459650    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lp9rq" podStartSLOduration=1.337974853 podStartE2EDuration="2.459634667s" totalImagesPullingTime="1.121659814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:55.458869351 +0000 UTC m=+23.270537564" watchObservedRunningTime="2026-11-03 13:08:55.459634667 +0000 UTC m=+23.271302850"
Nov 03 13:08:55.674563 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:55.674403    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fgqkh_44f5fcad-a6fb-4bd2-99aa-f4973415aafa/dns-node-resolver/0.log"
Nov 03 13:08:56.085434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.085403    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"]
Nov 03 13:08:56.087806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.087778    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.093538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.093513    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 13:08:56.093657 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.093513    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 13:08:56.093879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.093856    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 13:08:56.094409 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.094391    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x5kzr\""
Nov 03 13:08:56.094508 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.094471    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 13:08:56.094582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.094571    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 13:08:56.097375 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.097338    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"]
Nov 03 13:08:56.180729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.180686    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwqb7\" (UniqueName: \"kubernetes.io/projected/21f68575-8422-4984-ba8d-e4bc90daa416-kube-api-access-jwqb7\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.180914 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.180757    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21f68575-8422-4984-ba8d-e4bc90daa416-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.180914 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.180788    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21f68575-8422-4984-ba8d-e4bc90daa416-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.180914 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.180809    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21f68575-8422-4984-ba8d-e4bc90daa416-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.281880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.281845    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21f68575-8422-4984-ba8d-e4bc90daa416-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.282048 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.281887    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21f68575-8422-4984-ba8d-e4bc90daa416-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.282048 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.281913    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21f68575-8422-4984-ba8d-e4bc90daa416-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.282048 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.281973    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwqb7\" (UniqueName: \"kubernetes.io/projected/21f68575-8422-4984-ba8d-e4bc90daa416-kube-api-access-jwqb7\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.283898 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.283866    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21f68575-8422-4984-ba8d-e4bc90daa416-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.285423 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.285402    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21f68575-8422-4984-ba8d-e4bc90daa416-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.285557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.285447    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21f68575-8422-4984-ba8d-e4bc90daa416-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.296169 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.296141    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwqb7\" (UniqueName: \"kubernetes.io/projected/21f68575-8422-4984-ba8d-e4bc90daa416-kube-api-access-jwqb7\") pod \"prometheus-operator-9cd89b74c-m5zdq\" (UID: \"21f68575-8422-4984-ba8d-e4bc90daa416\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.402593 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.402518    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"
Nov 03 13:08:56.475854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.475830    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5dcf6c45d8-f5t74_39f044b1-cfab-41fb-a72a-819e22c431cf/registry/0.log"
Nov 03 13:08:56.696935 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:56.696890    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq"]
Nov 03 13:08:56.701102 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:08:56.701074    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f68575_8422_4984_ba8d_e4bc90daa416.slice/crio-fca7c73c0035a058bfcc2693d72f50f444f626816b1f0ac207b5c4de50fed98b WatchSource:0}: Error finding container fca7c73c0035a058bfcc2693d72f50f444f626816b1f0ac207b5c4de50fed98b: Status 404 returned error can't find the container with id fca7c73c0035a058bfcc2693d72f50f444f626816b1f0ac207b5c4de50fed98b
Nov 03 13:08:57.074013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:57.073941    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-88kk5_ba1ed75b-f38b-4589-87fa-0c8d22abe784/node-ca/0.log"
Nov 03 13:08:57.454611 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:57.454578    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc78ccb-6628n" event={"ID":"a0b80df0-a7c0-44e7-b7ee-31a565e1799b","Type":"ContainerStarted","Data":"b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32"}
Nov 03 13:08:57.456875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:57.456782    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq" event={"ID":"21f68575-8422-4984-ba8d-e4bc90daa416","Type":"ContainerStarted","Data":"fca7c73c0035a058bfcc2693d72f50f444f626816b1f0ac207b5c4de50fed98b"}
Nov 03 13:08:57.874550 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:57.874417    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7hff_e0bc4a29-7ff7-4b22-8f9f-56906b70ee41/serve-healthcheck-canary/0.log"
Nov 03 13:08:58.462720 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:58.462682    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq" event={"ID":"21f68575-8422-4984-ba8d-e4bc90daa416","Type":"ContainerStarted","Data":"c107b4a7ff38f165cc06fb18d98fc7cb79a7346c66f0a5cb2e37ef0dacc26a10"}
Nov 03 13:08:58.462909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:58.462748    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq" event={"ID":"21f68575-8422-4984-ba8d-e4bc90daa416","Type":"ContainerStarted","Data":"35feb1f62fff9a96fa391ded93980c0b03a43e157e38d70bc5bb8c5d9a5e9416"}
Nov 03 13:08:58.479492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:58.478713    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cc78ccb-6628n" podStartSLOduration=2.34025382 podStartE2EDuration="5.47847119s" totalImagesPullingTime="3.13821737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:57.473973844 +0000 UTC m=+25.285642048" watchObservedRunningTime="2026-11-03 13:08:58.47847119 +0000 UTC m=+26.290139379"
Nov 03 13:08:58.479492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:08:58.479218    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m5zdq" podStartSLOduration=1.077981723 podStartE2EDuration="2.479202662s" totalImagesPullingTime="1.401220939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:58.477896507 +0000 UTC m=+26.289564717" watchObservedRunningTime="2026-11-03 13:08:58.479202662 +0000 UTC m=+26.290870851"
Nov 03 13:09:00.464944 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.464910    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"]
Nov 03 13:09:00.470090 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.470067    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.474483 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.473223    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zjkhw"]
Nov 03 13:09:00.477112 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.477090    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.479667 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.479484    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"]
Nov 03 13:09:00.483528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.483467    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.483528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.483488    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"]
Nov 03 13:09:00.485470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.485263    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 13:09:00.485470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.485317    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 13:09:00.485470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.485442    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l6cwv\""
Nov 03 13:09:00.493755 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.493730    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 13:09:00.493892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.493858    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 13:09:00.504671 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.504649    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"]
Nov 03 13:09:00.512618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.511568    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 13:09:00.512618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.511659    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 13:09:00.512618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.511938    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9tvwf\""
Nov 03 13:09:00.512618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.511998    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 13:09:00.512618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.512025    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjbx2\""
Nov 03 13:09:00.512618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.512565    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 13:09:00.621847 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.621808    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfzf7\" (UniqueName: \"kubernetes.io/projected/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-api-access-rfzf7\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.621847 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.621846    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.622096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.621878    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvqt5\" (UniqueName: \"kubernetes.io/projected/0c682e8f-5f71-4a30-95f7-c86622eff068-kube-api-access-gvqt5\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.622096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.621949    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c682e8f-5f71-4a30-95f7-c86622eff068-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.622096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622012    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-wtmp\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.622259 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622100    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-tls\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.622259 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622151    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.622259 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622246    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.622443 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622274    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c682e8f-5f71-4a30-95f7-c86622eff068-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.622443 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622304    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c682e8f-5f71-4a30-95f7-c86622eff068-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.622443 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622408    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-sys\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.622443 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622430    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-metrics-client-ca\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.622648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622476    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-accelerators-collector-config\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.622648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622507    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.622648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622563    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-root\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.622648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622579    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z75s\" (UniqueName: \"kubernetes.io/projected/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-kube-api-access-6z75s\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.622648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622626    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.622648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622645    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-textfile\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.622871 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.622689    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.723779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.723693    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.723779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.723742    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.723779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.723772    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c682e8f-5f71-4a30-95f7-c86622eff068-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.724039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.723803    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c682e8f-5f71-4a30-95f7-c86622eff068-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.724039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.723883    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-sys\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.724039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.723930    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-metrics-client-ca\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.724039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.723953    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-accelerators-collector-config\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.724039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.723977    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.724039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.723993    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-root\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.724039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724015    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z75s\" (UniqueName: \"kubernetes.io/projected/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-kube-api-access-6z75s\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.724039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724036    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.724410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724055    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-textfile\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.724410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724202    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.724410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724240    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfzf7\" (UniqueName: \"kubernetes.io/projected/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-api-access-rfzf7\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.724410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724265    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.724410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724292    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqt5\" (UniqueName: \"kubernetes.io/projected/0c682e8f-5f71-4a30-95f7-c86622eff068-kube-api-access-gvqt5\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.724410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724322    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c682e8f-5f71-4a30-95f7-c86622eff068-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.724410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724369    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-wtmp\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.724410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724404    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-tls\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.725614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724890    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c682e8f-5f71-4a30-95f7-c86622eff068-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.725614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724928    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-sys\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.725614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.724980    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.725614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.725265    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-root\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.725614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.725286    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-textfile\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.725614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.725495    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.725614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.725503    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-metrics-client-ca\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.725614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.725572    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.726861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.726310    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-accelerators-collector-config\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.726861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.726468    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-wtmp\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.726861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.726849    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.727088 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.726966    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c682e8f-5f71-4a30-95f7-c86622eff068-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.727324 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.727297    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.727817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.727797    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.728915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.728886    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-node-exporter-tls\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.729294 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.729272    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c682e8f-5f71-4a30-95f7-c86622eff068-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.733915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.733894    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfzf7\" (UniqueName: \"kubernetes.io/projected/e439b615-5ffe-4341-b2d7-de50d3ff1eeb-kube-api-access-rfzf7\") pod \"kube-state-metrics-5dc8d7d89b-bl5ld\" (UID: \"e439b615-5ffe-4341-b2d7-de50d3ff1eeb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.734076 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.734058    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqt5\" (UniqueName: \"kubernetes.io/projected/0c682e8f-5f71-4a30-95f7-c86622eff068-kube-api-access-gvqt5\") pod \"openshift-state-metrics-7bd7489957-2vs99\" (UID: \"0c682e8f-5f71-4a30-95f7-c86622eff068\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.734254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.734234    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z75s\" (UniqueName: \"kubernetes.io/projected/bdb25057-8a32-4c8a-829a-a3b8b0c1487c-kube-api-access-6z75s\") pod \"node-exporter-zjkhw\" (UID: \"bdb25057-8a32-4c8a-829a-a3b8b0c1487c\") " pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.794538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.794496    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"
Nov 03 13:09:00.804994 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.804965    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zjkhw"
Nov 03 13:09:00.819043 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:00.819019    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"
Nov 03 13:09:00.822482 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:09:00.822454    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb25057_8a32_4c8a_829a_a3b8b0c1487c.slice/crio-b1db6bebbe9e31308ca0521d6c6b44690d8b9510e72a08621a86ce918c4e82dc WatchSource:0}: Error finding container b1db6bebbe9e31308ca0521d6c6b44690d8b9510e72a08621a86ce918c4e82dc: Status 404 returned error can't find the container with id b1db6bebbe9e31308ca0521d6c6b44690d8b9510e72a08621a86ce918c4e82dc
Nov 03 13:09:01.083661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.083523    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99"]
Nov 03 13:09:01.087371 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:09:01.087322    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c682e8f_5f71_4a30_95f7_c86622eff068.slice/crio-28f4bdb49b43cdcbb7404bb3a4e1a7b9c10fdabb7b9cf71df8ca48b4ba5f5c11 WatchSource:0}: Error finding container 28f4bdb49b43cdcbb7404bb3a4e1a7b9c10fdabb7b9cf71df8ca48b4ba5f5c11: Status 404 returned error can't find the container with id 28f4bdb49b43cdcbb7404bb3a4e1a7b9c10fdabb7b9cf71df8ca48b4ba5f5c11
Nov 03 13:09:01.117892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.117865    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld"]
Nov 03 13:09:01.123663 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:09:01.123634    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode439b615_5ffe_4341_b2d7_de50d3ff1eeb.slice/crio-1bcaf2cbbce46b614ac7fe10d2d06be7dba5d3ee158712f6b514044f4c015d7a WatchSource:0}: Error finding container 1bcaf2cbbce46b614ac7fe10d2d06be7dba5d3ee158712f6b514044f4c015d7a: Status 404 returned error can't find the container with id 1bcaf2cbbce46b614ac7fe10d2d06be7dba5d3ee158712f6b514044f4c015d7a
Nov 03 13:09:01.483268 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.483221    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld" event={"ID":"e439b615-5ffe-4341-b2d7-de50d3ff1eeb","Type":"ContainerStarted","Data":"1bcaf2cbbce46b614ac7fe10d2d06be7dba5d3ee158712f6b514044f4c015d7a"}
Nov 03 13:09:01.486047 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.486015    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99" event={"ID":"0c682e8f-5f71-4a30-95f7-c86622eff068","Type":"ContainerStarted","Data":"7e14eacf0fafde171f7c07255e32004f86e2d5eed328da2f61bc56c265224ea8"}
Nov 03 13:09:01.486198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.486078    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99" event={"ID":"0c682e8f-5f71-4a30-95f7-c86622eff068","Type":"ContainerStarted","Data":"c1c6ea7f2fe05e0c8e6dfd48696bf9d9a041590a66c8e6647f0481469365b80f"}
Nov 03 13:09:01.486198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.486106    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99" event={"ID":"0c682e8f-5f71-4a30-95f7-c86622eff068","Type":"ContainerStarted","Data":"28f4bdb49b43cdcbb7404bb3a4e1a7b9c10fdabb7b9cf71df8ca48b4ba5f5c11"}
Nov 03 13:09:01.487510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.487482    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zjkhw" event={"ID":"bdb25057-8a32-4c8a-829a-a3b8b0c1487c","Type":"ContainerStarted","Data":"b1db6bebbe9e31308ca0521d6c6b44690d8b9510e72a08621a86ce918c4e82dc"}
Nov 03 13:09:01.567719 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.567677    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 13:09:01.572376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.572318    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576246    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576292    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576369    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576392    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576398    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576448    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576469    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576523    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rpj69\""
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576580    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 13:09:01.577379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.576658    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 13:09:01.585695 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.585633    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.732904    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-web-config\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.732956    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733000    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-out\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733033    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733073    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733103    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733133    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733159    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733180    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sr76\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-kube-api-access-9sr76\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733217    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733242    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733274    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-volume\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.733609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.733296    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.833811 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.833761    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-out\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.833819    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.833858    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.833890    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.833918    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.833947    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.833971    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sr76\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-kube-api-access-9sr76\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.834008    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.834038    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.834066    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-volume\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.834081    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.834107    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-web-config\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.834393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.834143    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.836140 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.834748    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.836140 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.836028    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.840974 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.840952    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.841832 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.841812    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.843428 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.843014    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.845012 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.844987    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.851286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.851261    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sr76\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-kube-api-access-9sr76\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.861486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.861431    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.862182 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.862118    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.862733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.862706    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-volume\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.862972 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.862797    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.863728 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.863681    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-out\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.866328 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.866270    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-web-config\") pod \"alertmanager-main-0\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:01.903039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:01.902793    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:02.222873 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.222761    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 13:09:02.233878 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:09:02.233833    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd4777c_b0e3_41d5_ba28_ea0ec4c87c48.slice/crio-eda15ae96f5fdb91b981d54a8a8f6aca3acbf1d875617e9a378d908d115f9db5 WatchSource:0}: Error finding container eda15ae96f5fdb91b981d54a8a8f6aca3acbf1d875617e9a378d908d115f9db5: Status 404 returned error can't find the container with id eda15ae96f5fdb91b981d54a8a8f6aca3acbf1d875617e9a378d908d115f9db5
Nov 03 13:09:02.493123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.493089    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerStarted","Data":"eda15ae96f5fdb91b981d54a8a8f6aca3acbf1d875617e9a378d908d115f9db5"}
Nov 03 13:09:02.496242 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.496211    4447 generic.go:356] "Generic (PLEG): container finished" podID="bdb25057-8a32-4c8a-829a-a3b8b0c1487c" containerID="cf46cc15ea93fce4318ad35abfe29c9a08e4a9548271cc022eda9ba94a7b3811" exitCode=0
Nov 03 13:09:02.496371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.496259    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zjkhw" event={"ID":"bdb25057-8a32-4c8a-829a-a3b8b0c1487c","Type":"ContainerDied","Data":"cf46cc15ea93fce4318ad35abfe29c9a08e4a9548271cc022eda9ba94a7b3811"}
Nov 03 13:09:02.581502 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.581469    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-686567946f-4frpb"]
Nov 03 13:09:02.598806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.597902    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.598806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.598738    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-686567946f-4frpb"]
Nov 03 13:09:02.605184 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.605158    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 13:09:02.605626 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.605210    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-j0bcl5tlq9ff\""
Nov 03 13:09:02.605626 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.605262    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 13:09:02.605626 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.605270    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 13:09:02.605626 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.605368    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8zr5v\""
Nov 03 13:09:02.605626 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.605371    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 13:09:02.605626 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.605448    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 13:09:02.743277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.742925    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-tls\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.743277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.742970    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.743277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.742993    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f575583-222a-45d1-a7cc-b7dbdeae608c-tmp\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.743277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.743027    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.743277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.743043    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f575583-222a-45d1-a7cc-b7dbdeae608c-metrics-client-ca\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.743277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.743062    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.743277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.743091    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.743277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.743115    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxqp\" (UniqueName: \"kubernetes.io/projected/8f575583-222a-45d1-a7cc-b7dbdeae608c-kube-api-access-kdxqp\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.743277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.743131    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-grpc-tls\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.844046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844004    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.844218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844054    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f575583-222a-45d1-a7cc-b7dbdeae608c-metrics-client-ca\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.844218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844090    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.844218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844124    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.844218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844159    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxqp\" (UniqueName: \"kubernetes.io/projected/8f575583-222a-45d1-a7cc-b7dbdeae608c-kube-api-access-kdxqp\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.844542 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844518    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-grpc-tls\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.845082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844633    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-tls\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.845082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844669    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.845082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844705    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f575583-222a-45d1-a7cc-b7dbdeae608c-tmp\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.845082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.844940    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f575583-222a-45d1-a7cc-b7dbdeae608c-metrics-client-ca\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.845082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.845030    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f575583-222a-45d1-a7cc-b7dbdeae608c-tmp\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.848028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.847981    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.848611 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.848585    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.848914 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.848886    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-tls\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.849177 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.849135    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.849429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.849383    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.849689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.849584    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f575583-222a-45d1-a7cc-b7dbdeae608c-secret-grpc-tls\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:02.851800 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:02.851775    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxqp\" (UniqueName: \"kubernetes.io/projected/8f575583-222a-45d1-a7cc-b7dbdeae608c-kube-api-access-kdxqp\") pod \"thanos-querier-686567946f-4frpb\" (UID: \"8f575583-222a-45d1-a7cc-b7dbdeae608c\") " pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:03.020120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.020036    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:03.175809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.175774    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-78c86c496f-szc86"]
Nov 03 13:09:03.197570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.197539    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c86c496f-szc86"]
Nov 03 13:09:03.197731 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.197680    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.203581 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.203530    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 13:09:03.333129 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.333045    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-686567946f-4frpb"]
Nov 03 13:09:03.349774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.349744    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-serving-cert\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.349936 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.349858    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-oauth-config\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.349936 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.349901    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-oauth-serving-cert\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.350062 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.350016    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27w8\" (UniqueName: \"kubernetes.io/projected/2bfd1e52-e4dc-4e14-821e-d513105d1799-kube-api-access-x27w8\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.350124 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.350068    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-config\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.350124 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.350098    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-service-ca\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.350266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.350244    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfd1e52-e4dc-4e14-821e-d513105d1799-tmp\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.350309 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.350285    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-trusted-ca-bundle\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.355937 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:09:03.355885    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f575583_222a_45d1_a7cc_b7dbdeae608c.slice/crio-05c1c119895a2371948333159aa6be28559e3097607bd19c6c9147cc1f663c99 WatchSource:0}: Error finding container 05c1c119895a2371948333159aa6be28559e3097607bd19c6c9147cc1f663c99: Status 404 returned error can't find the container with id 05c1c119895a2371948333159aa6be28559e3097607bd19c6c9147cc1f663c99
Nov 03 13:09:03.439401 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:09:03.439282    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.451113    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-config\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.451179    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-service-ca\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.451239    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfd1e52-e4dc-4e14-821e-d513105d1799-tmp\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.451271    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-trusted-ca-bundle\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.451335    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-serving-cert\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.451419    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-oauth-config\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.451445    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-oauth-serving-cert\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.451464    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27w8\" (UniqueName: \"kubernetes.io/projected/2bfd1e52-e4dc-4e14-821e-d513105d1799-kube-api-access-x27w8\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.452429    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-config\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.452976    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-trusted-ca-bundle\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.453273    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfd1e52-e4dc-4e14-821e-d513105d1799-tmp\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.453336    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-service-ca\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.453985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.453922    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-oauth-serving-cert\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.462956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.462902    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27w8\" (UniqueName: \"kubernetes.io/projected/2bfd1e52-e4dc-4e14-821e-d513105d1799-kube-api-access-x27w8\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.467092 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.467062    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-serving-cert\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.469225 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.469202    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-oauth-config\") pod \"console-78c86c496f-szc86\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") " pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.504544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.503674    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99" event={"ID":"0c682e8f-5f71-4a30-95f7-c86622eff068","Type":"ContainerStarted","Data":"58a2d832f6fdb3857e94c66aef4538fa93ec6e1334a522c79b4c50b137ece314"}
Nov 03 13:09:03.508636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.508599    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zjkhw" event={"ID":"bdb25057-8a32-4c8a-829a-a3b8b0c1487c","Type":"ContainerStarted","Data":"19fc38ea12da2efcd589c9c81a3dcde2bd088482f798b63fbaf43e39f0c205f9"}
Nov 03 13:09:03.508783 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.508664    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zjkhw" event={"ID":"bdb25057-8a32-4c8a-829a-a3b8b0c1487c","Type":"ContainerStarted","Data":"90c969e259f906a0dd4f6207f0a9f92b50a5a15aaf1c2697f16e6ea2896e2f0c"}
Nov 03 13:09:03.510999 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.510971    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686567946f-4frpb" event={"ID":"8f575583-222a-45d1-a7cc-b7dbdeae608c","Type":"ContainerStarted","Data":"05c1c119895a2371948333159aa6be28559e3097607bd19c6c9147cc1f663c99"}
Nov 03 13:09:03.520523 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.520247    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:03.522475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.521436    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2vs99" podStartSLOduration=2.316847707 podStartE2EDuration="3.521420833s" totalImagesPullingTime="1.204573126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:03.520953704 +0000 UTC m=+31.332622200" watchObservedRunningTime="2026-11-03 13:09:03.521420833 +0000 UTC m=+31.333089023"
Nov 03 13:09:03.541129 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.539624    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zjkhw" podStartSLOduration=2.186696675 podStartE2EDuration="3.539607952s" totalImagesPullingTime="1.26099546s" totalInitContainerRuntime="91.915817ms" isStatefulPod=true podCreationTimestamp="2026-11-03 13:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:03.538884288 +0000 UTC m=+31.350552477" watchObservedRunningTime="2026-11-03 13:09:03.539607952 +0000 UTC m=+31.351276140"
Nov 03 13:09:03.745521 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.744759    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:09:03.745521 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.744816    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:09:03.754109 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.753882    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:09:03.863597 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:03.863479    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c86c496f-szc86"]
Nov 03 13:09:03.895332 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:09:03.895295    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfd1e52_e4dc_4e14_821e_d513105d1799.slice/crio-64a04cdbed1a75f7498ecc9250181aa0b13e5038b0be5be70ed7fb5b7ab2cd49 WatchSource:0}: Error finding container 64a04cdbed1a75f7498ecc9250181aa0b13e5038b0be5be70ed7fb5b7ab2cd49: Status 404 returned error can't find the container with id 64a04cdbed1a75f7498ecc9250181aa0b13e5038b0be5be70ed7fb5b7ab2cd49
Nov 03 13:09:04.519398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.519288    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld" event={"ID":"e439b615-5ffe-4341-b2d7-de50d3ff1eeb","Type":"ContainerStarted","Data":"3d6d0fcf496ffe719d6f82354bb9915e88d34f8bfaec4224aa8ba6be416d37f8"}
Nov 03 13:09:04.519398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.519382    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld" event={"ID":"e439b615-5ffe-4341-b2d7-de50d3ff1eeb","Type":"ContainerStarted","Data":"fc1829b63878a2409fc7ba070da583895b77a509b03e31e19b76996481b4367d"}
Nov 03 13:09:04.519861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.519405    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld" event={"ID":"e439b615-5ffe-4341-b2d7-de50d3ff1eeb","Type":"ContainerStarted","Data":"ecf42d3ee769a24cf440f10e1c2d05ba891d49469e6b14cf6399f1f8679b90dd"}
Nov 03 13:09:04.521772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.521747    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c86c496f-szc86" event={"ID":"2bfd1e52-e4dc-4e14-821e-d513105d1799","Type":"ContainerStarted","Data":"8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e"}
Nov 03 13:09:04.521904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.521792    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c86c496f-szc86" event={"ID":"2bfd1e52-e4dc-4e14-821e-d513105d1799","Type":"ContainerStarted","Data":"64a04cdbed1a75f7498ecc9250181aa0b13e5038b0be5be70ed7fb5b7ab2cd49"}
Nov 03 13:09:04.524302 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.524277    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerID="bd5b0e5f76fdd0e0f94f9ceb08a0110a09f9aa16d043607acb84639ff40045f0" exitCode=0
Nov 03 13:09:04.524441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.524400    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerDied","Data":"bd5b0e5f76fdd0e0f94f9ceb08a0110a09f9aa16d043607acb84639ff40045f0"}
Nov 03 13:09:04.530276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.530221    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:09:04.537741 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.537568    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bl5ld" podStartSLOduration=2.267717152 podStartE2EDuration="4.537554824s" totalImagesPullingTime="2.269837672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:04.535579739 +0000 UTC m=+32.347247960" watchObservedRunningTime="2026-11-03 13:09:04.537554824 +0000 UTC m=+32.349223012"
Nov 03 13:09:04.645564 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:04.645376    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-78c86c496f-szc86" podStartSLOduration=1.645338397 podStartE2EDuration="1.645338397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:04.64449126 +0000 UTC m=+32.456159486" watchObservedRunningTime="2026-11-03 13:09:04.645338397 +0000 UTC m=+32.457006587"
Nov 03 13:09:06.537370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:06.537315    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686567946f-4frpb" event={"ID":"8f575583-222a-45d1-a7cc-b7dbdeae608c","Type":"ContainerStarted","Data":"4a5ceb491db538089629bfc6b352ff75eb9fa94264231c4c1d7999115e9c202b"}
Nov 03 13:09:06.537796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:06.537447    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686567946f-4frpb" event={"ID":"8f575583-222a-45d1-a7cc-b7dbdeae608c","Type":"ContainerStarted","Data":"09b49611b8f2b7d19e6fbb2b024ab5d4da1de6c7a9e126cc4b7ccf66d801d4c5"}
Nov 03 13:09:06.764619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:06.764022    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ch6h8"
Nov 03 13:09:07.545817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:07.545782    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686567946f-4frpb" event={"ID":"8f575583-222a-45d1-a7cc-b7dbdeae608c","Type":"ContainerStarted","Data":"2a5675c3ce17f6079afba2316c1ea10214502d7ddac3a2b4161d525a3944474c"}
Nov 03 13:09:07.549542 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:07.549514    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerStarted","Data":"3d2de23808b26a3039d400820bdde017502a0f80d4b1aa9241682618593fa8bf"}
Nov 03 13:09:07.549689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:07.549586    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerStarted","Data":"216f28ea320ea2a38df7bf1eece0cdb9c5d053f80f92845bfb2f3c715e62c14d"}
Nov 03 13:09:07.549689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:07.549626    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerStarted","Data":"c0c82232d80719fadcfc761332ad7b019b9e1dcb827ffd5ef89b7667bcae02ff"}
Nov 03 13:09:07.549689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:07.549664    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerStarted","Data":"2e8c2f61dbde1792c84a7661ba566795781edb13729820799aa891dd59ade8b3"}
Nov 03 13:09:07.549813 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:07.549698    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerStarted","Data":"4b748283a02cf43821431e55eb2c2c7bad5d12514e568f17a37a9eface11ca75"}
Nov 03 13:09:08.557567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:08.557485    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerStarted","Data":"121de25770a061593a51ad3c8e9bfcaeea1ab092f3a8f48b5830a6ef60766626"}
Nov 03 13:09:08.560906 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:08.560860    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686567946f-4frpb" event={"ID":"8f575583-222a-45d1-a7cc-b7dbdeae608c","Type":"ContainerStarted","Data":"d1423cfd2ac3c72b39476da92c902ce6108394ac4e14e45cde1ab71ece002f27"}
Nov 03 13:09:08.561040 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:08.560937    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686567946f-4frpb" event={"ID":"8f575583-222a-45d1-a7cc-b7dbdeae608c","Type":"ContainerStarted","Data":"25812f243e66dee1308a7034e6f68627ca9a2675794a10925ec2287224edbd72"}
Nov 03 13:09:08.561422 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:08.561396    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686567946f-4frpb" event={"ID":"8f575583-222a-45d1-a7cc-b7dbdeae608c","Type":"ContainerStarted","Data":"3031fa7eaa984ad20bf59313dcba9d7a15162035210f6ef4b3e84f6cc2e2092d"}
Nov 03 13:09:08.561525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:08.561481    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:08.590333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:08.590279    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.704280859 podStartE2EDuration="7.590260998s" totalImagesPullingTime="4.861850636s" totalInitContainerRuntime="24.129503ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:08.587976199 +0000 UTC m=+36.399644445" watchObservedRunningTime="2026-11-03 13:09:08.590260998 +0000 UTC m=+36.401929186"
Nov 03 13:09:11.003036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.002229    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-686567946f-4frpb" podStartSLOduration=4.910452457 podStartE2EDuration="9.002208475s" totalImagesPullingTime="4.091756018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:08.617572055 +0000 UTC m=+36.429240248" watchObservedRunningTime="2026-11-03 13:09:11.002208475 +0000 UTC m=+38.813876663"
Nov 03 13:09:11.003036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.002647    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c86c496f-szc86"]
Nov 03 13:09:11.071705 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.071667    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f5859b879-cwsqj"]
Nov 03 13:09:11.075330 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.075303    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.090551 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.090518    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5859b879-cwsqj"]
Nov 03 13:09:11.242047 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.242015    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-oauth-serving-cert\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.242047 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.242051    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e13211c-540e-46b6-9620-3b571f02f514-tmp\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.242271 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.242086    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-serving-cert\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.242271 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.242241    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w7sz\" (UniqueName: \"kubernetes.io/projected/2e13211c-540e-46b6-9620-3b571f02f514-kube-api-access-6w7sz\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.242393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.242316    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-console-config\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.242455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.242435    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-oauth-config\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.242511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.242468    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-service-ca\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.242511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.242495    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-trusted-ca-bundle\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.344044 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.343956    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7sz\" (UniqueName: \"kubernetes.io/projected/2e13211c-540e-46b6-9620-3b571f02f514-kube-api-access-6w7sz\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.344044 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.344030    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-console-config\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.344279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.344099    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-oauth-config\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.344279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.344116    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-service-ca\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.344279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.344141    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-trusted-ca-bundle\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.344279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.344168    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-oauth-serving-cert\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.344279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.344198    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e13211c-540e-46b6-9620-3b571f02f514-tmp\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.344279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.344241    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-serving-cert\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.345055 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.345025    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-service-ca\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.345198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.345064    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-oauth-serving-cert\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.345198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.345168    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-trusted-ca-bundle\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.345323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.345288    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e13211c-540e-46b6-9620-3b571f02f514-tmp\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.345691 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.345666    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-console-config\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.347478 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.347458    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-serving-cert\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.347601 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.347585    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-oauth-config\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.354737 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.354716    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7sz\" (UniqueName: \"kubernetes.io/projected/2e13211c-540e-46b6-9620-3b571f02f514-kube-api-access-6w7sz\") pod \"console-5f5859b879-cwsqj\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") " pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.388134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.388101    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:11.683745 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:11.683713    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5859b879-cwsqj"]
Nov 03 13:09:11.695844 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:09:11.695814    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e13211c_540e_46b6_9620_3b571f02f514.slice/crio-aba93c38cbdd335c70f8bddea746ddb0bf68662cedfcd8de253fa1d473955ff7 WatchSource:0}: Error finding container aba93c38cbdd335c70f8bddea746ddb0bf68662cedfcd8de253fa1d473955ff7: Status 404 returned error can't find the container with id aba93c38cbdd335c70f8bddea746ddb0bf68662cedfcd8de253fa1d473955ff7
Nov 03 13:09:12.586976 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:12.586887    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j565n" event={"ID":"6da37367-7ecb-4213-a856-c7ae36609bae","Type":"ContainerStarted","Data":"3a29216ee961d888a14dbfa768c3ce87d057dd64892e2b50f84347898779803b"}
Nov 03 13:09:12.586976 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:12.586962    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:09:12.590099 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:12.590069    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5859b879-cwsqj" event={"ID":"2e13211c-540e-46b6-9620-3b571f02f514","Type":"ContainerStarted","Data":"7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7"}
Nov 03 13:09:12.590207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:12.590122    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5859b879-cwsqj" event={"ID":"2e13211c-540e-46b6-9620-3b571f02f514","Type":"ContainerStarted","Data":"aba93c38cbdd335c70f8bddea746ddb0bf68662cedfcd8de253fa1d473955ff7"}
Nov 03 13:09:12.593978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:12.593945    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-j565n"
Nov 03 13:09:12.605097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:12.605041    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-j565n" podStartSLOduration=1.3200134399999999 podStartE2EDuration="19.60502387s" totalImagesPullingTime="18.28501043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:12.603060376 +0000 UTC m=+40.414728582" watchObservedRunningTime="2026-11-03 13:09:12.60502387 +0000 UTC m=+40.416692059"
Nov 03 13:09:12.642441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:12.638653    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f5859b879-cwsqj" podStartSLOduration=1.6386314720000001 podStartE2EDuration="1.638631472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:12.637013882 +0000 UTC m=+40.448682089" watchObservedRunningTime="2026-11-03 13:09:12.638631472 +0000 UTC m=+40.450299661"
Nov 03 13:09:13.522253 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:13.522212    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:14.576853 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:14.576822    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-686567946f-4frpb"
Nov 03 13:09:15.451162 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:15.450466    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dcf6c45d8-f5t74"
Nov 03 13:09:21.388836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:21.388787    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:21.389370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:21.388859    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:21.393521 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:21.393505    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:21.638100 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:21.638065    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:09:21.684170 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:21.684142    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc78ccb-6628n"]
Nov 03 13:09:26.296830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.296793    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-f9xld-h6jgs"]
Nov 03 13:09:26.297238 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.296993    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-f9xld-h6jgs" podUID="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" containerName="insights-gathering" containerID="cri-o://ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c" gracePeriod=30
Nov 03 13:09:26.605731 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.605706    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:09:26.655294 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.655261    4447 generic.go:356] "Generic (PLEG): container finished" podID="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" containerID="ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c" exitCode=2
Nov 03 13:09:26.655467 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.655304    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f9xld-h6jgs"
Nov 03 13:09:26.655467 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.655325    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f9xld-h6jgs" event={"ID":"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f","Type":"ContainerDied","Data":"ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c"}
Nov 03 13:09:26.655467 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.655387    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f9xld-h6jgs" event={"ID":"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f","Type":"ContainerDied","Data":"5e7c3e44d06308063cee8fa279e17a7654386af7e2096777d3cff63364e7901b"}
Nov 03 13:09:26.655467 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.655420    4447 scope.go:122] "RemoveContainer" containerID="ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c"
Nov 03 13:09:26.663554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.663535    4447 scope.go:122] "RemoveContainer" containerID="ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c"
Nov 03 13:09:26.663810 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:09:26.663790    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c\": container with ID starting with ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c not found: ID does not exist" containerID="ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c"
Nov 03 13:09:26.663886 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.663829    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c"} err="failed to get container status \"ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c\": rpc error: code = NotFound desc = could not find container \"ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c\": container with ID starting with ce784a9d0e554641dd4ad52e9113513594d68a92ff6acc61e3ef7ad2266fcb0c not found: ID does not exist"
Nov 03 13:09:26.710865 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.710836    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle\") pod \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") "
Nov 03 13:09:26.710981 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.710880    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle\") pod \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") "
Nov 03 13:09:26.711019 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.710989    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path\") pod \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") "
Nov 03 13:09:26.711053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.711017    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb\" (UniqueName: \"kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb\") pod \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\" (UID: \"3eb93242-cc6c-4a41-bd2b-fa73aca1af1f\") "
Nov 03 13:09:26.711282 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.711259    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle" pod "3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" (UID: "3eb93242-cc6c-4a41-bd2b-fa73aca1af1f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:26.711282 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.711272    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle" pod "3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" (UID: "3eb93242-cc6c-4a41-bd2b-fa73aca1af1f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:26.711436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.711305    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path" pod "3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" (UID: "3eb93242-cc6c-4a41-bd2b-fa73aca1af1f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:26.713302 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.713274    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb" pod "3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" (UID: "3eb93242-cc6c-4a41-bd2b-fa73aca1af1f"). InnerVolumeSpecName "kube-api-access-4xwvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:26.812574 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.812545    4447 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-trusted-ca-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:26.812574 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.812570    4447 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-service-ca-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:26.812574 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.812579    4447 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-archives-path\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:26.812802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.812589    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xwvb\" (UniqueName: \"kubernetes.io/projected/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f-kube-api-access-4xwvb\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:26.974134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.974107    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-f9xld-h6jgs"]
Nov 03 13:09:26.977747 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:26.977727    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-f9xld-h6jgs"]
Nov 03 13:09:28.317816 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:28.317772    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" path="/var/lib/kubelet/pods/3eb93242-cc6c-4a41-bd2b-fa73aca1af1f/volumes"
Nov 03 13:09:33.409535 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:09:33.409495    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:09:36.024767 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.024728    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-78c86c496f-szc86" podUID="2bfd1e52-e4dc-4e14-821e-d513105d1799" containerName="console" containerID="cri-o://8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e" gracePeriod=15
Nov 03 13:09:36.301524 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.301458    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:36.396307 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396272    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-config\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-config\") pod \"2bfd1e52-e4dc-4e14-821e-d513105d1799\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") "
Nov 03 13:09:36.396505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396330    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-oauth-serving-cert\") pod \"2bfd1e52-e4dc-4e14-821e-d513105d1799\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") "
Nov 03 13:09:36.396505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396378    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-service-ca\") pod \"2bfd1e52-e4dc-4e14-821e-d513105d1799\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") "
Nov 03 13:09:36.396505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396403    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bfd1e52-e4dc-4e14-821e-d513105d1799-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfd1e52-e4dc-4e14-821e-d513105d1799-tmp\") pod \"2bfd1e52-e4dc-4e14-821e-d513105d1799\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") "
Nov 03 13:09:36.396505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396455    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bfd1e52-e4dc-4e14-821e-d513105d1799-kube-api-access-x27w8\" (UniqueName: \"kubernetes.io/projected/2bfd1e52-e4dc-4e14-821e-d513105d1799-kube-api-access-x27w8\") pod \"2bfd1e52-e4dc-4e14-821e-d513105d1799\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") "
Nov 03 13:09:36.396505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396488    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-trusted-ca-bundle\") pod \"2bfd1e52-e4dc-4e14-821e-d513105d1799\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") "
Nov 03 13:09:36.396757 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396544    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-serving-cert\") pod \"2bfd1e52-e4dc-4e14-821e-d513105d1799\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") "
Nov 03 13:09:36.396757 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396741    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfd1e52-e4dc-4e14-821e-d513105d1799-tmp" pod "2bfd1e52-e4dc-4e14-821e-d513105d1799" (UID: "2bfd1e52-e4dc-4e14-821e-d513105d1799"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:36.396857 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396762    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-oauth-config\") pod \"2bfd1e52-e4dc-4e14-821e-d513105d1799\" (UID: \"2bfd1e52-e4dc-4e14-821e-d513105d1799\") "
Nov 03 13:09:36.396857 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396771    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-config" pod "2bfd1e52-e4dc-4e14-821e-d513105d1799" (UID: "2bfd1e52-e4dc-4e14-821e-d513105d1799"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:36.396947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396906    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-service-ca" pod "2bfd1e52-e4dc-4e14-821e-d513105d1799" (UID: "2bfd1e52-e4dc-4e14-821e-d513105d1799"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:36.397006 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.396963    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-trusted-ca-bundle" pod "2bfd1e52-e4dc-4e14-821e-d513105d1799" (UID: "2bfd1e52-e4dc-4e14-821e-d513105d1799"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:36.397098 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.397078    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-oauth-serving-cert" pod "2bfd1e52-e4dc-4e14-821e-d513105d1799" (UID: "2bfd1e52-e4dc-4e14-821e-d513105d1799"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:36.397286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.397264    4447 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:36.397419 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.397378    4447 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfd1e52-e4dc-4e14-821e-d513105d1799-tmp\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:36.397419 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.397409    4447 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-trusted-ca-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:36.397544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.397425    4447 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:36.398867 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.398845    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-serving-cert" pod "2bfd1e52-e4dc-4e14-821e-d513105d1799" (UID: "2bfd1e52-e4dc-4e14-821e-d513105d1799"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:36.399171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.399143    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-oauth-config" pod "2bfd1e52-e4dc-4e14-821e-d513105d1799" (UID: "2bfd1e52-e4dc-4e14-821e-d513105d1799"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:36.399171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.399158    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfd1e52-e4dc-4e14-821e-d513105d1799-kube-api-access-x27w8" pod "2bfd1e52-e4dc-4e14-821e-d513105d1799" (UID: "2bfd1e52-e4dc-4e14-821e-d513105d1799"). InnerVolumeSpecName "kube-api-access-x27w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:36.498678 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.498640    4447 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-oauth-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:36.498678 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.498672    4447 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bfd1e52-e4dc-4e14-821e-d513105d1799-oauth-serving-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:36.498678 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.498681    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27w8\" (UniqueName: \"kubernetes.io/projected/2bfd1e52-e4dc-4e14-821e-d513105d1799-kube-api-access-x27w8\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:36.498909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.498691    4447 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bfd1e52-e4dc-4e14-821e-d513105d1799-console-serving-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:36.696365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.696304    4447 generic.go:356] "Generic (PLEG): container finished" podID="2bfd1e52-e4dc-4e14-821e-d513105d1799" containerID="8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e" exitCode=0
Nov 03 13:09:36.696554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.696379    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c86c496f-szc86"
Nov 03 13:09:36.696554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.696385    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c86c496f-szc86" event={"ID":"2bfd1e52-e4dc-4e14-821e-d513105d1799","Type":"ContainerDied","Data":"8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e"}
Nov 03 13:09:36.696554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.696434    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c86c496f-szc86" event={"ID":"2bfd1e52-e4dc-4e14-821e-d513105d1799","Type":"ContainerDied","Data":"64a04cdbed1a75f7498ecc9250181aa0b13e5038b0be5be70ed7fb5b7ab2cd49"}
Nov 03 13:09:36.696554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.696455    4447 scope.go:122] "RemoveContainer" containerID="8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e"
Nov 03 13:09:36.705811 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.705796    4447 scope.go:122] "RemoveContainer" containerID="8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e"
Nov 03 13:09:36.706085 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:09:36.706066    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e\": container with ID starting with 8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e not found: ID does not exist" containerID="8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e"
Nov 03 13:09:36.706132 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.706094    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e"} err="failed to get container status \"8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e\": rpc error: code = NotFound desc = could not find container \"8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e\": container with ID starting with 8a76c3693e0f4142b0bae24685b4a330a9e8805ef71ec7cd26600ae7c346831e not found: ID does not exist"
Nov 03 13:09:36.716643 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.716620    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c86c496f-szc86"]
Nov 03 13:09:36.722396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:36.722226    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78c86c496f-szc86"]
Nov 03 13:09:38.318065 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:38.318033    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bfd1e52-e4dc-4e14-821e-d513105d1799" path="/var/lib/kubelet/pods/2bfd1e52-e4dc-4e14-821e-d513105d1799/volumes"
Nov 03 13:09:46.704785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:46.704721    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-cc78ccb-6628n" podUID="a0b80df0-a7c0-44e7-b7ee-31a565e1799b" containerName="console" containerID="cri-o://b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32" gracePeriod=15
Nov 03 13:09:46.985011 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:46.984955    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:09:47.090487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090452    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-serving-cert\") pod \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") "
Nov 03 13:09:47.090487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090493    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-config\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-config\") pod \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") "
Nov 03 13:09:47.090724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090536    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-tmp\") pod \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") "
Nov 03 13:09:47.090724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090559    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-oauth-config\") pod \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") "
Nov 03 13:09:47.090724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090606    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-oauth-serving-cert\") pod \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") "
Nov 03 13:09:47.090724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090631    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-kube-api-access-nlmrw\" (UniqueName: \"kubernetes.io/projected/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-kube-api-access-nlmrw\") pod \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") "
Nov 03 13:09:47.090724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090658    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-service-ca\") pod \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\" (UID: \"a0b80df0-a7c0-44e7-b7ee-31a565e1799b\") "
Nov 03 13:09:47.090975 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090855    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-tmp" pod "a0b80df0-a7c0-44e7-b7ee-31a565e1799b" (UID: "a0b80df0-a7c0-44e7-b7ee-31a565e1799b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:47.090975 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090955    4447 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-tmp\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:47.090975 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090961    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-config" pod "a0b80df0-a7c0-44e7-b7ee-31a565e1799b" (UID: "a0b80df0-a7c0-44e7-b7ee-31a565e1799b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:47.091118 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.090997    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-oauth-serving-cert" pod "a0b80df0-a7c0-44e7-b7ee-31a565e1799b" (UID: "a0b80df0-a7c0-44e7-b7ee-31a565e1799b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:47.091177 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.091161    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-service-ca" pod "a0b80df0-a7c0-44e7-b7ee-31a565e1799b" (UID: "a0b80df0-a7c0-44e7-b7ee-31a565e1799b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:47.092757 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.092724    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-serving-cert" pod "a0b80df0-a7c0-44e7-b7ee-31a565e1799b" (UID: "a0b80df0-a7c0-44e7-b7ee-31a565e1799b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:47.092869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.092803    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-kube-api-access-nlmrw" pod "a0b80df0-a7c0-44e7-b7ee-31a565e1799b" (UID: "a0b80df0-a7c0-44e7-b7ee-31a565e1799b"). InnerVolumeSpecName "kube-api-access-nlmrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:47.092869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.092811    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-oauth-config" pod "a0b80df0-a7c0-44e7-b7ee-31a565e1799b" (UID: "a0b80df0-a7c0-44e7-b7ee-31a565e1799b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:47.191980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.191949    4447 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-serving-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:47.191980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.191976    4447 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:47.191980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.191986    4447 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-console-oauth-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:47.192219 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.191996    4447 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-oauth-serving-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:47.192219 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.192005    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlmrw\" (UniqueName: \"kubernetes.io/projected/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-kube-api-access-nlmrw\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:47.192219 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.192019    4447 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b80df0-a7c0-44e7-b7ee-31a565e1799b-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:47.742252 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.742216    4447 generic.go:356] "Generic (PLEG): container finished" podID="a0b80df0-a7c0-44e7-b7ee-31a565e1799b" containerID="b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32" exitCode=0
Nov 03 13:09:47.742661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.742267    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc78ccb-6628n"
Nov 03 13:09:47.742661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.742299    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc78ccb-6628n" event={"ID":"a0b80df0-a7c0-44e7-b7ee-31a565e1799b","Type":"ContainerDied","Data":"b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32"}
Nov 03 13:09:47.742661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.742360    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc78ccb-6628n" event={"ID":"a0b80df0-a7c0-44e7-b7ee-31a565e1799b","Type":"ContainerDied","Data":"424fdf925caf6b07c7b302dd47688980f0763c50da21e7f2abefd71c30d24637"}
Nov 03 13:09:47.742661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.742380    4447 scope.go:122] "RemoveContainer" containerID="b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32"
Nov 03 13:09:47.750770 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.750754    4447 scope.go:122] "RemoveContainer" containerID="b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32"
Nov 03 13:09:47.751051 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:09:47.751033    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32\": container with ID starting with b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32 not found: ID does not exist" containerID="b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32"
Nov 03 13:09:47.751130 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.751060    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32"} err="failed to get container status \"b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32\": rpc error: code = NotFound desc = could not find container \"b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32\": container with ID starting with b67f60b70f4ad935c39d8f426174c0e6a474b68dd08bb9cbdaf10308c1e59e32 not found: ID does not exist"
Nov 03 13:09:47.764431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.764409    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc78ccb-6628n"]
Nov 03 13:09:47.766601 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:47.766580    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cc78ccb-6628n"]
Nov 03 13:09:48.317128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:48.317096    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b80df0-a7c0-44e7-b7ee-31a565e1799b" path="/var/lib/kubelet/pods/a0b80df0-a7c0-44e7-b7ee-31a565e1799b/volumes"
Nov 03 13:09:50.271141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.271101    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 13:09:50.271570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.271448    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="alertmanager" containerID="cri-o://4b748283a02cf43821431e55eb2c2c7bad5d12514e568f17a37a9eface11ca75" gracePeriod=120
Nov 03 13:09:50.271570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.271478    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy-metric" containerID="cri-o://3d2de23808b26a3039d400820bdde017502a0f80d4b1aa9241682618593fa8bf" gracePeriod=120
Nov 03 13:09:50.271570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.271507    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="config-reloader" containerID="cri-o://2e8c2f61dbde1792c84a7661ba566795781edb13729820799aa891dd59ade8b3" gracePeriod=120
Nov 03 13:09:50.271570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.271517    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy" containerID="cri-o://216f28ea320ea2a38df7bf1eece0cdb9c5d053f80f92845bfb2f3c715e62c14d" gracePeriod=120
Nov 03 13:09:50.271784 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.271539    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy-web" containerID="cri-o://c0c82232d80719fadcfc761332ad7b019b9e1dcb827ffd5ef89b7667bcae02ff" gracePeriod=120
Nov 03 13:09:50.271784 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.271634    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="prom-label-proxy" containerID="cri-o://121de25770a061593a51ad3c8e9bfcaeea1ab092f3a8f48b5830a6ef60766626" gracePeriod=120
Nov 03 13:09:50.758453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758420    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerID="121de25770a061593a51ad3c8e9bfcaeea1ab092f3a8f48b5830a6ef60766626" exitCode=0
Nov 03 13:09:50.758453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758448    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerID="3d2de23808b26a3039d400820bdde017502a0f80d4b1aa9241682618593fa8bf" exitCode=0
Nov 03 13:09:50.758453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758455    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerID="216f28ea320ea2a38df7bf1eece0cdb9c5d053f80f92845bfb2f3c715e62c14d" exitCode=0
Nov 03 13:09:50.758453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758460    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerID="2e8c2f61dbde1792c84a7661ba566795781edb13729820799aa891dd59ade8b3" exitCode=0
Nov 03 13:09:50.758453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758465    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerID="4b748283a02cf43821431e55eb2c2c7bad5d12514e568f17a37a9eface11ca75" exitCode=0
Nov 03 13:09:50.758756 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758497    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerDied","Data":"121de25770a061593a51ad3c8e9bfcaeea1ab092f3a8f48b5830a6ef60766626"}
Nov 03 13:09:50.758756 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758558    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerDied","Data":"3d2de23808b26a3039d400820bdde017502a0f80d4b1aa9241682618593fa8bf"}
Nov 03 13:09:50.758756 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758584    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerDied","Data":"216f28ea320ea2a38df7bf1eece0cdb9c5d053f80f92845bfb2f3c715e62c14d"}
Nov 03 13:09:50.758756 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758608    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerDied","Data":"2e8c2f61dbde1792c84a7661ba566795781edb13729820799aa891dd59ade8b3"}
Nov 03 13:09:50.758756 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:50.758632    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerDied","Data":"4b748283a02cf43821431e55eb2c2c7bad5d12514e568f17a37a9eface11ca75"}
Nov 03 13:09:51.765952 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:51.765922    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerID="c0c82232d80719fadcfc761332ad7b019b9e1dcb827ffd5ef89b7667bcae02ff" exitCode=0
Nov 03 13:09:51.766319 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:51.765975    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerDied","Data":"c0c82232d80719fadcfc761332ad7b019b9e1dcb827ffd5ef89b7667bcae02ff"}
Nov 03 13:09:52.026216 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.026155    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:52.137960 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.137925    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-metrics-client-ca\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.137967    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-trusted-ca-bundle\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.137992    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-out\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-out\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138037    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-web\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138067    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-web-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-web-config\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138083    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-main-tls\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138111    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-cluster-tls-config\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138192    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-tls-assets\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-tls-assets\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138218    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138272    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-metric\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138309    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-main-db\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138360    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-kube-api-access-9sr76\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-kube-api-access-9sr76\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138397    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-volume\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-volume\") pod \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\" (UID: \"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48\") "
Nov 03 13:09:52.138512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138407    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-trusted-ca-bundle" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:52.138512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138493    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-metrics-client-ca" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:52.138963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138685    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-metrics-client-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.138963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.138707    4447 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.139483 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.139442    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-main-db" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:52.140891 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.140848    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-web" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:52.141432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.141405    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-out" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:52.141686 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.141657    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-tls-assets" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:52.141800 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.141711    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-volume" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:52.141920 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.141898    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-main-tls" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:52.142141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.142113    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-kube-api-access-9sr76" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "kube-api-access-9sr76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:52.142443 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.142421    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-metric" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:52.142638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.142613    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:52.146582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.146562    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-cluster-tls-config" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:52.153472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.153448    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-web-config" pod "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" (UID: "dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:52.239714 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239678    4447 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-out\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239714 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239707    4447 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239739    4447 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-web-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239750    4447 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-main-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239761    4447 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-cluster-tls-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239771    4447 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-tls-assets\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239780    4447 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239789    4447 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239797    4447 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-alertmanager-main-db\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239806    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sr76\" (UniqueName: \"kubernetes.io/projected/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-kube-api-access-9sr76\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.239958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.239814    4447 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:52.772733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.772700    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48","Type":"ContainerDied","Data":"eda15ae96f5fdb91b981d54a8a8f6aca3acbf1d875617e9a378d908d115f9db5"}
Nov 03 13:09:52.773135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.772750    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:52.773135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.772772    4447 scope.go:122] "RemoveContainer" containerID="121de25770a061593a51ad3c8e9bfcaeea1ab092f3a8f48b5830a6ef60766626"
Nov 03 13:09:52.780601 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.780582    4447 scope.go:122] "RemoveContainer" containerID="3d2de23808b26a3039d400820bdde017502a0f80d4b1aa9241682618593fa8bf"
Nov 03 13:09:52.787234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.787212    4447 scope.go:122] "RemoveContainer" containerID="216f28ea320ea2a38df7bf1eece0cdb9c5d053f80f92845bfb2f3c715e62c14d"
Nov 03 13:09:52.793976 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.793954    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 13:09:52.794558 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.794541    4447 scope.go:122] "RemoveContainer" containerID="c0c82232d80719fadcfc761332ad7b019b9e1dcb827ffd5ef89b7667bcae02ff"
Nov 03 13:09:52.801761 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.801742    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 13:09:52.802154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.802142    4447 scope.go:122] "RemoveContainer" containerID="2e8c2f61dbde1792c84a7661ba566795781edb13729820799aa891dd59ade8b3"
Nov 03 13:09:52.808859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.808843    4447 scope.go:122] "RemoveContainer" containerID="4b748283a02cf43821431e55eb2c2c7bad5d12514e568f17a37a9eface11ca75"
Nov 03 13:09:52.815875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.815858    4447 scope.go:122] "RemoveContainer" containerID="bd5b0e5f76fdd0e0f94f9ceb08a0110a09f9aa16d043607acb84639ff40045f0"
Nov 03 13:09:52.833958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.833936    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 13:09:52.834301 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834286    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="prom-label-proxy"
Nov 03 13:09:52.834363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834304    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="prom-label-proxy"
Nov 03 13:09:52.834363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834328    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy"
Nov 03 13:09:52.834363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834334    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy"
Nov 03 13:09:52.834363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834358    4447 cpu_manager.go:414] "containerMap: removing container" podUID="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" containerName="insights-gathering"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834365    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" containerName="insights-gathering"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834376    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="config-reloader"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834382    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="config-reloader"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834398    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="init-config-reloader"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834405    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="init-config-reloader"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834414    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="alertmanager"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834420    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="alertmanager"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834436    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy-web"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834442    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy-web"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834450    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy-metric"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834456    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy-metric"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834467    4447 cpu_manager.go:414] "containerMap: removing container" podUID="2bfd1e52-e4dc-4e14-821e-d513105d1799" containerName="console"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834473    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bfd1e52-e4dc-4e14-821e-d513105d1799" containerName="console"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834482    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a0b80df0-a7c0-44e7-b7ee-31a565e1799b" containerName="console"
Nov 03 13:09:52.834499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834488    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b80df0-a7c0-44e7-b7ee-31a565e1799b" containerName="console"
Nov 03 13:09:52.834947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834556    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="prom-label-proxy"
Nov 03 13:09:52.834947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834569    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy"
Nov 03 13:09:52.834947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834576    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb93242-cc6c-4a41-bd2b-fa73aca1af1f" containerName="insights-gathering"
Nov 03 13:09:52.834947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834583    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="config-reloader"
Nov 03 13:09:52.834947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834593    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="alertmanager"
Nov 03 13:09:52.834947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834601    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy-web"
Nov 03 13:09:52.834947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834607    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" containerName="kube-rbac-proxy-metric"
Nov 03 13:09:52.834947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834612    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bfd1e52-e4dc-4e14-821e-d513105d1799" containerName="console"
Nov 03 13:09:52.834947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.834617    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b80df0-a7c0-44e7-b7ee-31a565e1799b" containerName="console"
Nov 03 13:09:52.870198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.870171    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 13:09:52.870369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.870333    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:52.875932 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.875905    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 13:09:52.876038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.875928    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 13:09:52.876038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.875975    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 13:09:52.876139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.876069    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 13:09:52.876139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.876073    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 13:09:52.876139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.876081    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 13:09:52.876139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.876118    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 13:09:52.876308 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.876198    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 13:09:52.876308 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.876217    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rpj69\""
Nov 03 13:09:52.877283 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:52.877244    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 13:09:53.046300 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046210    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-config-volume\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046300 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046267    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046319    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395945ec-ad7b-446f-a8d9-87b611ba8523-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046460    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/395945ec-ad7b-446f-a8d9-87b611ba8523-config-out\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046492    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-web-config\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046539    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046566    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/395945ec-ad7b-446f-a8d9-87b611ba8523-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046586    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/395945ec-ad7b-446f-a8d9-87b611ba8523-tls-assets\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046649    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046682    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/395945ec-ad7b-446f-a8d9-87b611ba8523-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046702    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046741    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.046892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.046815    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpzrd\" (UniqueName: \"kubernetes.io/projected/395945ec-ad7b-446f-a8d9-87b611ba8523-kube-api-access-hpzrd\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.147685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147640    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-config-volume\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.147685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147691    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.147956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147709    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395945ec-ad7b-446f-a8d9-87b611ba8523-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.147956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147727    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/395945ec-ad7b-446f-a8d9-87b611ba8523-config-out\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.147956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147743    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-web-config\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.147956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147760    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.147956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147782    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/395945ec-ad7b-446f-a8d9-87b611ba8523-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.147956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147903    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/395945ec-ad7b-446f-a8d9-87b611ba8523-tls-assets\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.147956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147950    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.148296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.147990    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/395945ec-ad7b-446f-a8d9-87b611ba8523-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.148296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.148022    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.148296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.148052    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.148296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.148087    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpzrd\" (UniqueName: \"kubernetes.io/projected/395945ec-ad7b-446f-a8d9-87b611ba8523-kube-api-access-hpzrd\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.148296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.148253    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/395945ec-ad7b-446f-a8d9-87b611ba8523-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.148812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.148779    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395945ec-ad7b-446f-a8d9-87b611ba8523-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.148986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.148962    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/395945ec-ad7b-446f-a8d9-87b611ba8523-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.150969 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.150847    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-config-volume\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.150969 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.150948    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.150969 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.150960    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-web-config\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.151178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.151072    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.151251 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.151227    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.151472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.151448    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/395945ec-ad7b-446f-a8d9-87b611ba8523-config-out\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.151565 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.151543    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/395945ec-ad7b-446f-a8d9-87b611ba8523-tls-assets\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.151696 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.151677    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.152947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.152930    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/395945ec-ad7b-446f-a8d9-87b611ba8523-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.156904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.156887    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpzrd\" (UniqueName: \"kubernetes.io/projected/395945ec-ad7b-446f-a8d9-87b611ba8523-kube-api-access-hpzrd\") pod \"alertmanager-main-0\" (UID: \"395945ec-ad7b-446f-a8d9-87b611ba8523\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.302533 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.302434    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 13:09:53.616010 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.615982    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 13:09:53.618236 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:09:53.618207    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395945ec_ad7b_446f_a8d9_87b611ba8523.slice/crio-ed72d709da4a801d8518586625cac7c7c8ba7931c2b8a0776ee7bf302104fd6e WatchSource:0}: Error finding container ed72d709da4a801d8518586625cac7c7c8ba7931c2b8a0776ee7bf302104fd6e: Status 404 returned error can't find the container with id ed72d709da4a801d8518586625cac7c7c8ba7931c2b8a0776ee7bf302104fd6e
Nov 03 13:09:53.778926 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.778890    4447 generic.go:356] "Generic (PLEG): container finished" podID="395945ec-ad7b-446f-a8d9-87b611ba8523" containerID="da3adef60b7b756aaeedac8f31d2837665a28d4b84c368e88bfca1753ab01314" exitCode=0
Nov 03 13:09:53.779301 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.778976    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395945ec-ad7b-446f-a8d9-87b611ba8523","Type":"ContainerDied","Data":"da3adef60b7b756aaeedac8f31d2837665a28d4b84c368e88bfca1753ab01314"}
Nov 03 13:09:53.779301 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:53.779046    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395945ec-ad7b-446f-a8d9-87b611ba8523","Type":"ContainerStarted","Data":"ed72d709da4a801d8518586625cac7c7c8ba7931c2b8a0776ee7bf302104fd6e"}
Nov 03 13:09:54.317752 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:54.317717    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48" path="/var/lib/kubelet/pods/dfd4777c-b0e3-41d5-ba28-ea0ec4c87c48/volumes"
Nov 03 13:09:54.786333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:54.786292    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395945ec-ad7b-446f-a8d9-87b611ba8523","Type":"ContainerStarted","Data":"76140ff4aefd64c2f42f473a06f362f341e618042a80aafab86b82723498c721"}
Nov 03 13:09:54.786923 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:54.786897    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395945ec-ad7b-446f-a8d9-87b611ba8523","Type":"ContainerStarted","Data":"98beee08a6c062163bfcaf1c5f19330c0784e7a4e9346e5447c602a8385e388a"}
Nov 03 13:09:54.787024 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:54.786955    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395945ec-ad7b-446f-a8d9-87b611ba8523","Type":"ContainerStarted","Data":"3a39ae26b8836d7bba1a59443fff800f7ed220a82b486f47b26d6ae04a5e79fc"}
Nov 03 13:09:54.787024 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:54.786993    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395945ec-ad7b-446f-a8d9-87b611ba8523","Type":"ContainerStarted","Data":"fa54faaf23a564954744c8324717a0dac7bc88862d85b4105232d122436ed85e"}
Nov 03 13:09:54.787123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:54.787028    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395945ec-ad7b-446f-a8d9-87b611ba8523","Type":"ContainerStarted","Data":"763cc960d9f462780db0831e0b4dd2d6c9188215809206d02692a4dd4746df30"}
Nov 03 13:09:54.787123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:54.787063    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395945ec-ad7b-446f-a8d9-87b611ba8523","Type":"ContainerStarted","Data":"a14d75693af178989c90741b700debdfbd678e1a066dc30c85e83a381e32f1cc"}
Nov 03 13:09:54.823291 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:54.823234    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.80963934 podStartE2EDuration="2.823213195s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.573855ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:54.821224348 +0000 UTC m=+82.632892557" watchObservedRunningTime="2026-11-03 13:09:54.823213195 +0000 UTC m=+82.634881386"
Nov 03 13:09:57.394400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.394364    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5855b954c4-knw4g"]
Nov 03 13:09:57.396546 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.396528    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.414418 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.414395    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5855b954c4-knw4g"]
Nov 03 13:09:57.485857 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.485814    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-service-ca\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.486022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.485878    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a51f6038-9849-410d-b30f-593873df2834-tmp\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.486022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.485952    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-trusted-ca-bundle\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.486022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.485992    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2gr\" (UniqueName: \"kubernetes.io/projected/a51f6038-9849-410d-b30f-593873df2834-kube-api-access-ws2gr\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.486022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.486016    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-console-config\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.486161 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.486105    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-oauth-config\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.486161 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.486149    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-serving-cert\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.486216 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.486181    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-oauth-serving-cert\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.586782 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.586748    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2gr\" (UniqueName: \"kubernetes.io/projected/a51f6038-9849-410d-b30f-593873df2834-kube-api-access-ws2gr\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.586782 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.586786    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-console-config\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.586951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.586804    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-oauth-config\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.586951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.586847    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-serving-cert\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.586951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.586878    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-oauth-serving-cert\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.586951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.586908    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-service-ca\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.587152 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.586968    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a51f6038-9849-410d-b30f-593873df2834-tmp\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.587152 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.586998    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-trusted-ca-bundle\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.587524 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.587490    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a51f6038-9849-410d-b30f-593873df2834-tmp\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.587794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.587767    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-service-ca\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.587881 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.587767    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-oauth-serving-cert\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.587948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.587875    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-console-config\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.587948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.587912    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-trusted-ca-bundle\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.589439 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.589421    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-serving-cert\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.589519 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.589477    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-oauth-config\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.594768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.594745    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2gr\" (UniqueName: \"kubernetes.io/projected/a51f6038-9849-410d-b30f-593873df2834-kube-api-access-ws2gr\") pod \"console-5855b954c4-knw4g\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") " pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.706890 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.706855    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:09:57.980172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:57.980144    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5855b954c4-knw4g"]
Nov 03 13:09:57.982574 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:09:57.982545    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51f6038_9849_410d_b30f_593873df2834.slice/crio-c9d998bb6a12a424245a98fed5b2d31f6fa151976455c13f1733dcac75d79dc6 WatchSource:0}: Error finding container c9d998bb6a12a424245a98fed5b2d31f6fa151976455c13f1733dcac75d79dc6: Status 404 returned error can't find the container with id c9d998bb6a12a424245a98fed5b2d31f6fa151976455c13f1733dcac75d79dc6
Nov 03 13:09:58.806662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:58.806627    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5855b954c4-knw4g" event={"ID":"a51f6038-9849-410d-b30f-593873df2834","Type":"ContainerStarted","Data":"8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6"}
Nov 03 13:09:58.807060 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:58.806672    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5855b954c4-knw4g" event={"ID":"a51f6038-9849-410d-b30f-593873df2834","Type":"ContainerStarted","Data":"c9d998bb6a12a424245a98fed5b2d31f6fa151976455c13f1733dcac75d79dc6"}
Nov 03 13:09:58.823852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:09:58.823803    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5855b954c4-knw4g" podStartSLOduration=1.8237897969999999 podStartE2EDuration="1.823789797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:58.823645295 +0000 UTC m=+86.635313503" watchObservedRunningTime="2026-11-03 13:09:58.823789797 +0000 UTC m=+86.635457985"
Nov 03 13:10:03.410272 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:10:03.410234    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:10:07.707288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:07.707250    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:10:07.707736 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:07.707366    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:10:07.712060 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:07.712042    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:10:07.846359 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:07.846315    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:10:07.884154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:07.884119    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f5859b879-cwsqj"]
Nov 03 13:10:19.891503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:19.891467    4447 generic.go:356] "Generic (PLEG): container finished" podID="f5ef0a05-3328-401e-946c-cc7b1582b15c" containerID="83686f3d151d98f57bd8c7cbc3d43815daa57a2a41e51817457a725cbc5a0e6f" exitCode=0
Nov 03 13:10:19.891503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:19.891506    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p694g-sqtcz" event={"ID":"f5ef0a05-3328-401e-946c-cc7b1582b15c","Type":"ContainerDied","Data":"83686f3d151d98f57bd8c7cbc3d43815daa57a2a41e51817457a725cbc5a0e6f"}
Nov 03 13:10:21.015576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.015553    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:10:21.096733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.096693    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-trusted-ca-bundle\") pod \"f5ef0a05-3328-401e-946c-cc7b1582b15c\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") "
Nov 03 13:10:21.096733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.096735    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-service-ca-bundle\") pod \"f5ef0a05-3328-401e-946c-cc7b1582b15c\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") "
Nov 03 13:10:21.096942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.096798    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ef0a05-3328-401e-946c-cc7b1582b15c-kube-api-access-m58gv\" (UniqueName: \"kubernetes.io/projected/f5ef0a05-3328-401e-946c-cc7b1582b15c-kube-api-access-m58gv\") pod \"f5ef0a05-3328-401e-946c-cc7b1582b15c\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") "
Nov 03 13:10:21.096942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.096830    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ef0a05-3328-401e-946c-cc7b1582b15c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ef0a05-3328-401e-946c-cc7b1582b15c-archives-path\") pod \"f5ef0a05-3328-401e-946c-cc7b1582b15c\" (UID: \"f5ef0a05-3328-401e-946c-cc7b1582b15c\") "
Nov 03 13:10:21.097180 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.097149    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-service-ca-bundle" pod "f5ef0a05-3328-401e-946c-cc7b1582b15c" (UID: "f5ef0a05-3328-401e-946c-cc7b1582b15c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:10:21.097249 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.097158    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-trusted-ca-bundle" pod "f5ef0a05-3328-401e-946c-cc7b1582b15c" (UID: "f5ef0a05-3328-401e-946c-cc7b1582b15c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:10:21.097249 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.097164    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ef0a05-3328-401e-946c-cc7b1582b15c-archives-path" pod "f5ef0a05-3328-401e-946c-cc7b1582b15c" (UID: "f5ef0a05-3328-401e-946c-cc7b1582b15c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:10:21.098922 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.098907    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ef0a05-3328-401e-946c-cc7b1582b15c-kube-api-access-m58gv" pod "f5ef0a05-3328-401e-946c-cc7b1582b15c" (UID: "f5ef0a05-3328-401e-946c-cc7b1582b15c"). InnerVolumeSpecName "kube-api-access-m58gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:10:21.198140 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.198091    4447 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-trusted-ca-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:21.198140 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.198137    4447 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef0a05-3328-401e-946c-cc7b1582b15c-service-ca-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:21.198140 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.198151    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m58gv\" (UniqueName: \"kubernetes.io/projected/f5ef0a05-3328-401e-946c-cc7b1582b15c-kube-api-access-m58gv\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:21.198414 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.198167    4447 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ef0a05-3328-401e-946c-cc7b1582b15c-archives-path\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:21.899432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.899394    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p694g-sqtcz" event={"ID":"f5ef0a05-3328-401e-946c-cc7b1582b15c","Type":"ContainerDied","Data":"721763a0989d01cc4d28fe714acd6b646a63ae21a75f83b4f62d8630bacdf8dc"}
Nov 03 13:10:21.899432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.899439    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721763a0989d01cc4d28fe714acd6b646a63ae21a75f83b4f62d8630bacdf8dc"
Nov 03 13:10:21.899657 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:21.899437    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p694g-sqtcz"
Nov 03 13:10:32.904679 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:32.904640    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f5859b879-cwsqj" podUID="2e13211c-540e-46b6-9620-3b571f02f514" containerName="console" containerID="cri-o://7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7" gracePeriod=15
Nov 03 13:10:33.147138 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.147115    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:10:33.303517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.303475    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e13211c-540e-46b6-9620-3b571f02f514-kube-api-access-6w7sz\" (UniqueName: \"kubernetes.io/projected/2e13211c-540e-46b6-9620-3b571f02f514-kube-api-access-6w7sz\") pod \"2e13211c-540e-46b6-9620-3b571f02f514\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") "
Nov 03 13:10:33.303517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.303522    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-oauth-serving-cert\") pod \"2e13211c-540e-46b6-9620-3b571f02f514\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") "
Nov 03 13:10:33.303765 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.303542    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-service-ca\") pod \"2e13211c-540e-46b6-9620-3b571f02f514\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") "
Nov 03 13:10:33.303765 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.303570    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-serving-cert\") pod \"2e13211c-540e-46b6-9620-3b571f02f514\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") "
Nov 03 13:10:33.303765 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.303615    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-oauth-config\") pod \"2e13211c-540e-46b6-9620-3b571f02f514\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") "
Nov 03 13:10:33.303765 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.303646    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-trusted-ca-bundle\") pod \"2e13211c-540e-46b6-9620-3b571f02f514\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") "
Nov 03 13:10:33.303765 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.303676    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e13211c-540e-46b6-9620-3b571f02f514-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e13211c-540e-46b6-9620-3b571f02f514-tmp\") pod \"2e13211c-540e-46b6-9620-3b571f02f514\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") "
Nov 03 13:10:33.304028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.303828    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-console-config\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-console-config\") pod \"2e13211c-540e-46b6-9620-3b571f02f514\" (UID: \"2e13211c-540e-46b6-9620-3b571f02f514\") "
Nov 03 13:10:33.304028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.303960    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-service-ca" pod "2e13211c-540e-46b6-9620-3b571f02f514" (UID: "2e13211c-540e-46b6-9620-3b571f02f514"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:10:33.304028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.304005    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-oauth-serving-cert" pod "2e13211c-540e-46b6-9620-3b571f02f514" (UID: "2e13211c-540e-46b6-9620-3b571f02f514"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:10:33.304028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.304015    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e13211c-540e-46b6-9620-3b571f02f514-tmp" pod "2e13211c-540e-46b6-9620-3b571f02f514" (UID: "2e13211c-540e-46b6-9620-3b571f02f514"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:10:33.304286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.304267    4447 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-oauth-serving-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:33.304362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.304292    4447 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:33.304362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.304299    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-trusted-ca-bundle" pod "2e13211c-540e-46b6-9620-3b571f02f514" (UID: "2e13211c-540e-46b6-9620-3b571f02f514"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:10:33.304362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.304306    4447 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e13211c-540e-46b6-9620-3b571f02f514-tmp\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:33.304485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.304379    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-console-config" pod "2e13211c-540e-46b6-9620-3b571f02f514" (UID: "2e13211c-540e-46b6-9620-3b571f02f514"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:10:33.305824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.305801    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-serving-cert" pod "2e13211c-540e-46b6-9620-3b571f02f514" (UID: "2e13211c-540e-46b6-9620-3b571f02f514"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:10:33.306237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.306208    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-oauth-config" pod "2e13211c-540e-46b6-9620-3b571f02f514" (UID: "2e13211c-540e-46b6-9620-3b571f02f514"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:10:33.306237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.306211    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e13211c-540e-46b6-9620-3b571f02f514-kube-api-access-6w7sz" pod "2e13211c-540e-46b6-9620-3b571f02f514" (UID: "2e13211c-540e-46b6-9620-3b571f02f514"). InnerVolumeSpecName "kube-api-access-6w7sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:10:33.406185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.406144    4447 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-oauth-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:33.406185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.406175    4447 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-trusted-ca-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:33.406185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.406188    4447 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e13211c-540e-46b6-9620-3b571f02f514-console-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:33.406472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.406202    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w7sz\" (UniqueName: \"kubernetes.io/projected/2e13211c-540e-46b6-9620-3b571f02f514-kube-api-access-6w7sz\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:33.406472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.406217    4447 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e13211c-540e-46b6-9620-3b571f02f514-console-serving-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:33.409541 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:10:33.409510    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:10:33.946142 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.946109    4447 generic.go:356] "Generic (PLEG): container finished" podID="2e13211c-540e-46b6-9620-3b571f02f514" containerID="7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7" exitCode=0
Nov 03 13:10:33.946612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.946154    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5859b879-cwsqj"
Nov 03 13:10:33.946612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.946186    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5859b879-cwsqj" event={"ID":"2e13211c-540e-46b6-9620-3b571f02f514","Type":"ContainerDied","Data":"7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7"}
Nov 03 13:10:33.946612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.946225    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5859b879-cwsqj" event={"ID":"2e13211c-540e-46b6-9620-3b571f02f514","Type":"ContainerDied","Data":"aba93c38cbdd335c70f8bddea746ddb0bf68662cedfcd8de253fa1d473955ff7"}
Nov 03 13:10:33.946612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.946246    4447 scope.go:122] "RemoveContainer" containerID="7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7"
Nov 03 13:10:33.956128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.956112    4447 scope.go:122] "RemoveContainer" containerID="7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7"
Nov 03 13:10:33.956410 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:10:33.956391    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7\": container with ID starting with 7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7 not found: ID does not exist" containerID="7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7"
Nov 03 13:10:33.956506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.956420    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7"} err="failed to get container status \"7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7\": rpc error: code = NotFound desc = could not find container \"7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7\": container with ID starting with 7f9b98890ec2959aa223e8426705de6cdc5901781a83508d01fdbe13af8b34b7 not found: ID does not exist"
Nov 03 13:10:33.967892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.967862    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f5859b879-cwsqj"]
Nov 03 13:10:33.971919 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:33.971898    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f5859b879-cwsqj"]
Nov 03 13:10:34.317364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:34.317280    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e13211c-540e-46b6-9620-3b571f02f514" path="/var/lib/kubelet/pods/2e13211c-540e-46b6-9620-3b571f02f514/volumes"
Nov 03 13:10:41.075385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.075336    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-964cc69f9-gxnx5"]
Nov 03 13:10:41.075772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.075693    4447 cpu_manager.go:414] "containerMap: removing container" podUID="2e13211c-540e-46b6-9620-3b571f02f514" containerName="console"
Nov 03 13:10:41.075772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.075704    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e13211c-540e-46b6-9620-3b571f02f514" containerName="console"
Nov 03 13:10:41.075772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.075736    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f5ef0a05-3328-401e-946c-cc7b1582b15c" containerName="insights-gathering"
Nov 03 13:10:41.075772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.075742    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ef0a05-3328-401e-946c-cc7b1582b15c" containerName="insights-gathering"
Nov 03 13:10:41.075896 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.075792    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ef0a05-3328-401e-946c-cc7b1582b15c" containerName="insights-gathering"
Nov 03 13:10:41.075896 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.075800    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e13211c-540e-46b6-9620-3b571f02f514" containerName="console"
Nov 03 13:10:41.078466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.078447    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.091651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.091626    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-964cc69f9-gxnx5"]
Nov 03 13:10:41.171981 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.171942    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-console-config\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.172127 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.171988    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-trusted-ca-bundle\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.172127 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.172018    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25c9c616-da13-4317-8b5c-78584a815f02-tmp\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.172226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.172117    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25c9c616-da13-4317-8b5c-78584a815f02-console-serving-cert\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.172226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.172146    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-service-ca\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.172226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.172212    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25c9c616-da13-4317-8b5c-78584a815f02-console-oauth-config\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.172312 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.172241    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5zw\" (UniqueName: \"kubernetes.io/projected/25c9c616-da13-4317-8b5c-78584a815f02-kube-api-access-rw5zw\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.172370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.172319    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-oauth-serving-cert\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.273247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.273210    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-oauth-serving-cert\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.273432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.273250    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-console-config\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.273432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.273292    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-trusted-ca-bundle\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.273432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.273323    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25c9c616-da13-4317-8b5c-78584a815f02-tmp\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.273432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.273375    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25c9c616-da13-4317-8b5c-78584a815f02-console-serving-cert\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.273432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.273398    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-service-ca\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.273699 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.273450    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25c9c616-da13-4317-8b5c-78584a815f02-console-oauth-config\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.273699 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.273482    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5zw\" (UniqueName: \"kubernetes.io/projected/25c9c616-da13-4317-8b5c-78584a815f02-kube-api-access-rw5zw\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.273822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.273797    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25c9c616-da13-4317-8b5c-78584a815f02-tmp\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.274076 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.274056    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-oauth-serving-cert\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.274159 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.274108    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-console-config\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.274327 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.274304    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-trusted-ca-bundle\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.274417 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.274375    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25c9c616-da13-4317-8b5c-78584a815f02-service-ca\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.275979 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.275955    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25c9c616-da13-4317-8b5c-78584a815f02-console-serving-cert\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.276073 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.275955    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25c9c616-da13-4317-8b5c-78584a815f02-console-oauth-config\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.280711 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.280692    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5zw\" (UniqueName: \"kubernetes.io/projected/25c9c616-da13-4317-8b5c-78584a815f02-kube-api-access-rw5zw\") pod \"console-964cc69f9-gxnx5\" (UID: \"25c9c616-da13-4317-8b5c-78584a815f02\") " pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.389162 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.389075    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:41.660612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.660536    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-964cc69f9-gxnx5"]
Nov 03 13:10:41.664045 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:10:41.664019    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c9c616_da13_4317_8b5c_78584a815f02.slice/crio-94f91711baacb190edcb3b8d627d112be93c963e1e60a7416d0845796bbb4353 WatchSource:0}: Error finding container 94f91711baacb190edcb3b8d627d112be93c963e1e60a7416d0845796bbb4353: Status 404 returned error can't find the container with id 94f91711baacb190edcb3b8d627d112be93c963e1e60a7416d0845796bbb4353
Nov 03 13:10:41.979867 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.979832    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-964cc69f9-gxnx5" event={"ID":"25c9c616-da13-4317-8b5c-78584a815f02","Type":"ContainerStarted","Data":"ceb87dc80dae692d4514a19eb29d31e3d463b8cfa40c3312892f4174ad963071"}
Nov 03 13:10:41.980049 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.979885    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-964cc69f9-gxnx5" event={"ID":"25c9c616-da13-4317-8b5c-78584a815f02","Type":"ContainerStarted","Data":"94f91711baacb190edcb3b8d627d112be93c963e1e60a7416d0845796bbb4353"}
Nov 03 13:10:41.997107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:41.997058    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-964cc69f9-gxnx5" podStartSLOduration=0.997045595 podStartE2EDuration="997.045595ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:41.995731078 +0000 UTC m=+129.807399274" watchObservedRunningTime="2026-11-03 13:10:41.997045595 +0000 UTC m=+129.808713783"
Nov 03 13:10:51.389431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:51.389330    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:51.389431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:51.389411    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:51.394088 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:51.394062    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:52.024633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:52.024601    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-964cc69f9-gxnx5"
Nov 03 13:10:52.070681 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:10:52.070650    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5855b954c4-knw4g"]
Nov 03 13:11:03.410073 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:11:03.410032    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:11:17.091350 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.091282    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5855b954c4-knw4g" podUID="a51f6038-9849-410d-b30f-593873df2834" containerName="console" containerID="cri-o://8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6" gracePeriod=15
Nov 03 13:11:17.327195 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.327166    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:11:17.477400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477339    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-trusted-ca-bundle\") pod \"a51f6038-9849-410d-b30f-593873df2834\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") "
Nov 03 13:11:17.477596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477413    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-oauth-config\") pod \"a51f6038-9849-410d-b30f-593873df2834\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") "
Nov 03 13:11:17.477596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477508    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-service-ca\") pod \"a51f6038-9849-410d-b30f-593873df2834\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") "
Nov 03 13:11:17.477596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477554    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-console-config\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-console-config\") pod \"a51f6038-9849-410d-b30f-593873df2834\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") "
Nov 03 13:11:17.477596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477574    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51f6038-9849-410d-b30f-593873df2834-kube-api-access-ws2gr\" (UniqueName: \"kubernetes.io/projected/a51f6038-9849-410d-b30f-593873df2834-kube-api-access-ws2gr\") pod \"a51f6038-9849-410d-b30f-593873df2834\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") "
Nov 03 13:11:17.477822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477720    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51f6038-9849-410d-b30f-593873df2834-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a51f6038-9849-410d-b30f-593873df2834-tmp\") pod \"a51f6038-9849-410d-b30f-593873df2834\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") "
Nov 03 13:11:17.477822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477728    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-trusted-ca-bundle" pod "a51f6038-9849-410d-b30f-593873df2834" (UID: "a51f6038-9849-410d-b30f-593873df2834"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:11:17.477822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477796    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-serving-cert\") pod \"a51f6038-9849-410d-b30f-593873df2834\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") "
Nov 03 13:11:17.477982 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477842    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-oauth-serving-cert\") pod \"a51f6038-9849-410d-b30f-593873df2834\" (UID: \"a51f6038-9849-410d-b30f-593873df2834\") "
Nov 03 13:11:17.477982 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477922    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-console-config" pod "a51f6038-9849-410d-b30f-593873df2834" (UID: "a51f6038-9849-410d-b30f-593873df2834"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:11:17.477982 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.477926    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-service-ca" pod "a51f6038-9849-410d-b30f-593873df2834" (UID: "a51f6038-9849-410d-b30f-593873df2834"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:11:17.478134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.478061    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51f6038-9849-410d-b30f-593873df2834-tmp" pod "a51f6038-9849-410d-b30f-593873df2834" (UID: "a51f6038-9849-410d-b30f-593873df2834"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:11:17.478207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.478187    4447 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-trusted-ca-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:17.478274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.478214    4447 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:17.478274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.478229    4447 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-console-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:17.478274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.478241    4447 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a51f6038-9849-410d-b30f-593873df2834-tmp\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:17.478460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.478361    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-oauth-serving-cert" pod "a51f6038-9849-410d-b30f-593873df2834" (UID: "a51f6038-9849-410d-b30f-593873df2834"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:11:17.479713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.479691    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-oauth-config" pod "a51f6038-9849-410d-b30f-593873df2834" (UID: "a51f6038-9849-410d-b30f-593873df2834"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:11:17.479713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.479697    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51f6038-9849-410d-b30f-593873df2834-kube-api-access-ws2gr" pod "a51f6038-9849-410d-b30f-593873df2834" (UID: "a51f6038-9849-410d-b30f-593873df2834"). InnerVolumeSpecName "kube-api-access-ws2gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:11:17.479859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.479836    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-serving-cert" pod "a51f6038-9849-410d-b30f-593873df2834" (UID: "a51f6038-9849-410d-b30f-593873df2834"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:11:17.579103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.579068    4447 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-oauth-config\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:17.579103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.579098    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws2gr\" (UniqueName: \"kubernetes.io/projected/a51f6038-9849-410d-b30f-593873df2834-kube-api-access-ws2gr\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:17.579103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.579109    4447 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a51f6038-9849-410d-b30f-593873df2834-console-serving-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:17.579308 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:17.579117    4447 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a51f6038-9849-410d-b30f-593873df2834-oauth-serving-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:18.126008 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.125970    4447 generic.go:356] "Generic (PLEG): container finished" podID="a51f6038-9849-410d-b30f-593873df2834" containerID="8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6" exitCode=0
Nov 03 13:11:18.126008 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.126007    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5855b954c4-knw4g" event={"ID":"a51f6038-9849-410d-b30f-593873df2834","Type":"ContainerDied","Data":"8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6"}
Nov 03 13:11:18.126591 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.126032    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5855b954c4-knw4g"
Nov 03 13:11:18.126591 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.126067    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5855b954c4-knw4g" event={"ID":"a51f6038-9849-410d-b30f-593873df2834","Type":"ContainerDied","Data":"c9d998bb6a12a424245a98fed5b2d31f6fa151976455c13f1733dcac75d79dc6"}
Nov 03 13:11:18.126591 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.126087    4447 scope.go:122] "RemoveContainer" containerID="8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6"
Nov 03 13:11:18.135044 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.135026    4447 scope.go:122] "RemoveContainer" containerID="8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6"
Nov 03 13:11:18.135299 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:11:18.135279    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6\": container with ID starting with 8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6 not found: ID does not exist" containerID="8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6"
Nov 03 13:11:18.135397 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.135311    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6"} err="failed to get container status \"8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6\": rpc error: code = NotFound desc = could not find container \"8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6\": container with ID starting with 8cadeac5947630873fb83e71afe089702537c5ddffec59a02f85773e248b54e6 not found: ID does not exist"
Nov 03 13:11:18.145790 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.145767    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5855b954c4-knw4g"]
Nov 03 13:11:18.149823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.149802    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5855b954c4-knw4g"]
Nov 03 13:11:18.317067 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:11:18.317037    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a51f6038-9849-410d-b30f-593873df2834" path="/var/lib/kubelet/pods/a51f6038-9849-410d-b30f-593873df2834/volumes"
Nov 03 13:11:33.409768 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:11:33.409729    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:12:03.410065 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:12:03.410029    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:12:33.409954 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:12:33.409871    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:13:01.784425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.784388    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"]
Nov 03 13:13:01.786796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.784740    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a51f6038-9849-410d-b30f-593873df2834" containerName="console"
Nov 03 13:13:01.786796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.784751    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51f6038-9849-410d-b30f-593873df2834" containerName="console"
Nov 03 13:13:01.786796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.784806    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a51f6038-9849-410d-b30f-593873df2834" containerName="console"
Nov 03 13:13:01.787671 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.787655    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:01.790982 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.790962    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-pcv4d\""
Nov 03 13:13:01.791250 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.791232    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 13:13:01.791353 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.791328    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 13:13:01.795928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.795904    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"]
Nov 03 13:13:01.820898 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.820859    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8vr\" (UniqueName: \"kubernetes.io/projected/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-kube-api-access-5v8vr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:01.821067 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.820925    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:01.821067 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.820992    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:01.883487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.883457    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"]
Nov 03 13:13:01.886750 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.886733    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:01.889522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.889496    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-vb6px\""
Nov 03 13:13:01.895306 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.895281    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"]
Nov 03 13:13:01.922170 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.922134    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:01.922338 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.922182    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:01.922338 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.922233    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:01.922338 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.922272    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x5mb\" (UniqueName: \"kubernetes.io/projected/1d854b63-1e8f-438b-a86c-08d5eada9b87-kube-api-access-7x5mb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:01.922528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.922382    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8vr\" (UniqueName: \"kubernetes.io/projected/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-kube-api-access-5v8vr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:01.922528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.922417    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:01.922528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.922510    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:01.922624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.922533    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:01.930695 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:01.930675    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8vr\" (UniqueName: \"kubernetes.io/projected/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-kube-api-access-5v8vr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:02.023494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.023460    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:02.023674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.023511    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x5mb\" (UniqueName: \"kubernetes.io/projected/1d854b63-1e8f-438b-a86c-08d5eada9b87-kube-api-access-7x5mb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:02.023674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.023551    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:02.023833 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.023814    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:02.023889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.023873    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:02.030701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.030679    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x5mb\" (UniqueName: \"kubernetes.io/projected/1d854b63-1e8f-438b-a86c-08d5eada9b87-kube-api-access-7x5mb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:02.104545 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.104461    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:02.202823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.202789    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:02.377201 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.377124    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"]
Nov 03 13:13:02.380061 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:13:02.380033    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d7e875_ad46_4aa3_8f00_2edf8e609fff.slice/crio-326fa2dee1742aaca7783b64ab38fb52efca2d6ecc128fb4a4b1d0aae458b9ee WatchSource:0}: Error finding container 326fa2dee1742aaca7783b64ab38fb52efca2d6ecc128fb4a4b1d0aae458b9ee: Status 404 returned error can't find the container with id 326fa2dee1742aaca7783b64ab38fb52efca2d6ecc128fb4a4b1d0aae458b9ee
Nov 03 13:13:02.480852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.480822    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"]
Nov 03 13:13:02.486608 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:13:02.486572    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d854b63_1e8f_438b_a86c_08d5eada9b87.slice/crio-25d6ed898b49b3a911bf4dea93758bc3043ae312c2e491f3940452a0a19913a2 WatchSource:0}: Error finding container 25d6ed898b49b3a911bf4dea93758bc3043ae312c2e491f3940452a0a19913a2: Status 404 returned error can't find the container with id 25d6ed898b49b3a911bf4dea93758bc3043ae312c2e491f3940452a0a19913a2
Nov 03 13:13:02.520861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.520822    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh" event={"ID":"1d854b63-1e8f-438b-a86c-08d5eada9b87","Type":"ContainerStarted","Data":"25d6ed898b49b3a911bf4dea93758bc3043ae312c2e491f3940452a0a19913a2"}
Nov 03 13:13:02.521878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:02.521854    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8" event={"ID":"d5d7e875-ad46-4aa3-8f00-2edf8e609fff","Type":"ContainerStarted","Data":"326fa2dee1742aaca7783b64ab38fb52efca2d6ecc128fb4a4b1d0aae458b9ee"}
Nov 03 13:13:03.416641 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:13:03.416605    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:13:03.703381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.703327    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"]
Nov 03 13:13:03.707557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.707532    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:03.714091 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.713548    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-sdwpw\""
Nov 03 13:13:03.718383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.715511    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"]
Nov 03 13:13:03.838468 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.838432    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:03.838638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.838482    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrgr\" (UniqueName: \"kubernetes.io/projected/eb4af09e-b40a-471e-a82c-96abd2158615-kube-api-access-jqrgr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:03.838638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.838600    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:03.939740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.939701    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrgr\" (UniqueName: \"kubernetes.io/projected/eb4af09e-b40a-471e-a82c-96abd2158615-kube-api-access-jqrgr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:03.939924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.939749    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:03.939924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.939820    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:03.940257 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.940233    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:03.940303 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.940246    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:03.948967 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:03.948936    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrgr\" (UniqueName: \"kubernetes.io/projected/eb4af09e-b40a-471e-a82c-96abd2158615-kube-api-access-jqrgr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:04.029689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:04.029607    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:04.319562 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:04.319453    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"]
Nov 03 13:13:04.323387 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:13:04.323362    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4af09e_b40a_471e_a82c_96abd2158615.slice/crio-af22573730e22c6f0a294ab3f6e6022fbf18e11a53d6595960a95c4e29c97c99 WatchSource:0}: Error finding container af22573730e22c6f0a294ab3f6e6022fbf18e11a53d6595960a95c4e29c97c99: Status 404 returned error can't find the container with id af22573730e22c6f0a294ab3f6e6022fbf18e11a53d6595960a95c4e29c97c99
Nov 03 13:13:04.534888 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:04.534844    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn" event={"ID":"eb4af09e-b40a-471e-a82c-96abd2158615","Type":"ContainerStarted","Data":"af22573730e22c6f0a294ab3f6e6022fbf18e11a53d6595960a95c4e29c97c99"}
Nov 03 13:13:08.726362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.722871    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"]
Nov 03 13:13:08.740459 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.728942    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.744535 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.743801    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"]
Nov 03 13:13:08.788396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.787241    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.788396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.787293    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.788396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.787434    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfvd\" (UniqueName: \"kubernetes.io/projected/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-kube-api-access-qdfvd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.803429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.803400    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"]
Nov 03 13:13:08.807381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.806914    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:08.816460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.815876    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"]
Nov 03 13:13:08.835644 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.835613    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"]
Nov 03 13:13:08.839150 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.839128    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:08.848401 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.848378    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"]
Nov 03 13:13:08.867916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.867890    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9wg28\""
Nov 03 13:13:08.888550 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.888521    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:08.888698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.888557    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjnql\" (UniqueName: \"kubernetes.io/projected/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-kube-api-access-kjnql\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:08.888698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.888667    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.888804 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.888713    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:08.888804 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.888759    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:08.888804 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.888789    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.889096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.889077    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.889150 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.889094    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.889246 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.889225    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhs94\" (UniqueName: \"kubernetes.io/projected/c2cebb8b-2fd3-47f0-8e81-342094772686-kube-api-access-dhs94\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:08.889307 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.889264    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:08.889477 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.889451    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfvd\" (UniqueName: \"kubernetes.io/projected/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-kube-api-access-qdfvd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.906663 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.906639    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfvd\" (UniqueName: \"kubernetes.io/projected/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-kube-api-access-qdfvd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:08.990045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.989950    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:08.990045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.990003    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjnql\" (UniqueName: \"kubernetes.io/projected/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-kube-api-access-kjnql\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:08.990434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.990402    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:08.990548 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.990409    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:08.990548 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.990461    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:08.990664 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.990569    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhs94\" (UniqueName: \"kubernetes.io/projected/c2cebb8b-2fd3-47f0-8e81-342094772686-kube-api-access-dhs94\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:08.990664 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.990610    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:08.990664 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.990631    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:08.990844 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.990818    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:08.990965 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.990939    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:08.998079 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:08.998050    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhs94\" (UniqueName: \"kubernetes.io/projected/c2cebb8b-2fd3-47f0-8e81-342094772686-kube-api-access-dhs94\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:09.001480 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.001453    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjnql\" (UniqueName: \"kubernetes.io/projected/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-kube-api-access-kjnql\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:09.062218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.062181    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:09.142738 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.142706    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:09.173223 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.172973    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:09.361939 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.361901    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"]
Nov 03 13:13:09.366697 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:13:09.366659    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f69df3d_62c1_4209_845b_e7e00cdfdcbb.slice/crio-5d3d5eae7841d6dd110a3a2c5a58890908740e10dfbc1d258c4f401f9555e580 WatchSource:0}: Error finding container 5d3d5eae7841d6dd110a3a2c5a58890908740e10dfbc1d258c4f401f9555e580: Status 404 returned error can't find the container with id 5d3d5eae7841d6dd110a3a2c5a58890908740e10dfbc1d258c4f401f9555e580
Nov 03 13:13:09.437109 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.437077    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"]
Nov 03 13:13:09.440982 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:13:09.440941    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cebb8b_2fd3_47f0_8e81_342094772686.slice/crio-921b37c38aceb811d3ba7c01092d3d80a18363a56a985b2b66c9cc82d2785abb WatchSource:0}: Error finding container 921b37c38aceb811d3ba7c01092d3d80a18363a56a985b2b66c9cc82d2785abb: Status 404 returned error can't find the container with id 921b37c38aceb811d3ba7c01092d3d80a18363a56a985b2b66c9cc82d2785abb
Nov 03 13:13:09.469465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.469438    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"]
Nov 03 13:13:09.473282 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:13:09.473247    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cff7a0_c174_4365_8aaa_ba3fbdf81ce5.slice/crio-4c498c82e05af08d6c9c53aa40caf66819cfa91ad6130571683fd36ea0e61990 WatchSource:0}: Error finding container 4c498c82e05af08d6c9c53aa40caf66819cfa91ad6130571683fd36ea0e61990: Status 404 returned error can't find the container with id 4c498c82e05af08d6c9c53aa40caf66819cfa91ad6130571683fd36ea0e61990
Nov 03 13:13:09.563755 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.563668    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x" event={"ID":"6f69df3d-62c1-4209-845b-e7e00cdfdcbb","Type":"ContainerStarted","Data":"5d3d5eae7841d6dd110a3a2c5a58890908740e10dfbc1d258c4f401f9555e580"}
Nov 03 13:13:09.566751 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.566721    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l" event={"ID":"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5","Type":"ContainerStarted","Data":"4c498c82e05af08d6c9c53aa40caf66819cfa91ad6130571683fd36ea0e61990"}
Nov 03 13:13:09.568161 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:09.568131    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv" event={"ID":"c2cebb8b-2fd3-47f0-8e81-342094772686","Type":"ContainerStarted","Data":"921b37c38aceb811d3ba7c01092d3d80a18363a56a985b2b66c9cc82d2785abb"}
Nov 03 13:13:10.575515 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.575473    4447 generic.go:356] "Generic (PLEG): container finished" podID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerID="6cfff197a7e3224c3e5901915b0593ba9cb5ed3878bc0c4db3c919a5ab80efa3" exitCode=0
Nov 03 13:13:10.575980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.575560    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8" event={"ID":"d5d7e875-ad46-4aa3-8f00-2edf8e609fff","Type":"ContainerDied","Data":"6cfff197a7e3224c3e5901915b0593ba9cb5ed3878bc0c4db3c919a5ab80efa3"}
Nov 03 13:13:10.577327 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.577304    4447 generic.go:356] "Generic (PLEG): container finished" podID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerID="110000e7ef0958c51da33081ed4d00a69e2841615855056028d85f0e4d6d0a9e" exitCode=0
Nov 03 13:13:10.577429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.577382    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x" event={"ID":"6f69df3d-62c1-4209-845b-e7e00cdfdcbb","Type":"ContainerDied","Data":"110000e7ef0958c51da33081ed4d00a69e2841615855056028d85f0e4d6d0a9e"}
Nov 03 13:13:10.579571 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.579550    4447 generic.go:356] "Generic (PLEG): container finished" podID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerID="4c2104e5f002a939c6c88ccd5fbd9f99d1b26de4eb3d7dfd694f58da51f3ee96" exitCode=0
Nov 03 13:13:10.579677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.579628    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l" event={"ID":"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5","Type":"ContainerDied","Data":"4c2104e5f002a939c6c88ccd5fbd9f99d1b26de4eb3d7dfd694f58da51f3ee96"}
Nov 03 13:13:10.581321 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.581297    4447 generic.go:356] "Generic (PLEG): container finished" podID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerID="f8d95a4768ebde1601c2aface19c448348196b0cfba647e80fe023d84030f041" exitCode=0
Nov 03 13:13:10.584437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.584412    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh" event={"ID":"1d854b63-1e8f-438b-a86c-08d5eada9b87","Type":"ContainerDied","Data":"f8d95a4768ebde1601c2aface19c448348196b0cfba647e80fe023d84030f041"}
Nov 03 13:13:10.584542 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.584443    4447 generic.go:356] "Generic (PLEG): container finished" podID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerID="ac4cb7b951e4c2df759b44b04d92ff08f2322c6d480bb57ec7476053ae76219a" exitCode=0
Nov 03 13:13:10.584542 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:10.584461    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv" event={"ID":"c2cebb8b-2fd3-47f0-8e81-342094772686","Type":"ContainerDied","Data":"ac4cb7b951e4c2df759b44b04d92ff08f2322c6d480bb57ec7476053ae76219a"}
Nov 03 13:13:11.591254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:11.591217    4447 generic.go:356] "Generic (PLEG): container finished" podID="eb4af09e-b40a-471e-a82c-96abd2158615" containerID="4dedaaec9cefcef596933dd698231bc82438a8f06517408b743c325754a5193e" exitCode=0
Nov 03 13:13:11.591694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:11.591292    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn" event={"ID":"eb4af09e-b40a-471e-a82c-96abd2158615","Type":"ContainerDied","Data":"4dedaaec9cefcef596933dd698231bc82438a8f06517408b743c325754a5193e"}
Nov 03 13:13:12.597491 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:12.597455    4447 generic.go:356] "Generic (PLEG): container finished" podID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerID="eadaa7a7b8d4a58a18a1f875dfd77b0b38adb999b096d782bfed642ece19c513" exitCode=0
Nov 03 13:13:12.597930 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:12.597539    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x" event={"ID":"6f69df3d-62c1-4209-845b-e7e00cdfdcbb","Type":"ContainerDied","Data":"eadaa7a7b8d4a58a18a1f875dfd77b0b38adb999b096d782bfed642ece19c513"}
Nov 03 13:13:12.599468 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:12.599386    4447 generic.go:356] "Generic (PLEG): container finished" podID="eb4af09e-b40a-471e-a82c-96abd2158615" containerID="44810e0b94a87b8cf1016707fe0a82700d2b3654ed7a8405c88648728e1357b2" exitCode=0
Nov 03 13:13:12.599468 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:12.599414    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn" event={"ID":"eb4af09e-b40a-471e-a82c-96abd2158615","Type":"ContainerDied","Data":"44810e0b94a87b8cf1016707fe0a82700d2b3654ed7a8405c88648728e1357b2"}
Nov 03 13:13:12.601410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:12.601392    4447 generic.go:356] "Generic (PLEG): container finished" podID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerID="c5a17f93e6c92a7ee26a0049738f37d9f932a39f3b1511686ca58c1e13accc81" exitCode=0
Nov 03 13:13:12.601497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:12.601484    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh" event={"ID":"1d854b63-1e8f-438b-a86c-08d5eada9b87","Type":"ContainerDied","Data":"c5a17f93e6c92a7ee26a0049738f37d9f932a39f3b1511686ca58c1e13accc81"}
Nov 03 13:13:12.603294 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:12.603273    4447 generic.go:356] "Generic (PLEG): container finished" podID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerID="c660e04bda76412555735e145e7346e5dd8596f632036592695ee47f30ce20df" exitCode=0
Nov 03 13:13:12.603381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:12.603321    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8" event={"ID":"d5d7e875-ad46-4aa3-8f00-2edf8e609fff","Type":"ContainerDied","Data":"c660e04bda76412555735e145e7346e5dd8596f632036592695ee47f30ce20df"}
Nov 03 13:13:16.643959 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:16.643918    4447 generic.go:356] "Generic (PLEG): container finished" podID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerID="4c8d2363a431e8ecb7af810026ec32275a7d7f777271e2796f4d8ca5868e44ba" exitCode=0
Nov 03 13:13:16.644544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:16.644061    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv" event={"ID":"c2cebb8b-2fd3-47f0-8e81-342094772686","Type":"ContainerDied","Data":"4c8d2363a431e8ecb7af810026ec32275a7d7f777271e2796f4d8ca5868e44ba"}
Nov 03 13:13:23.692397 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:23.692362    4447 generic.go:356] "Generic (PLEG): container finished" podID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerID="0b96c3367509c478937b0eac99bee33d57cdd9cb37d7bc9a04e572e80ee9f54e" exitCode=0
Nov 03 13:13:23.692855 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:23.692396    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh" event={"ID":"1d854b63-1e8f-438b-a86c-08d5eada9b87","Type":"ContainerDied","Data":"0b96c3367509c478937b0eac99bee33d57cdd9cb37d7bc9a04e572e80ee9f54e"}
Nov 03 13:13:23.694714 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:23.694690    4447 generic.go:356] "Generic (PLEG): container finished" podID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerID="67fa498915a8c7894d7269aa0893d60230fadf3022da643ebcb00b25a0e84e73" exitCode=0
Nov 03 13:13:23.694834 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:23.694763    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv" event={"ID":"c2cebb8b-2fd3-47f0-8e81-342094772686","Type":"ContainerDied","Data":"67fa498915a8c7894d7269aa0893d60230fadf3022da643ebcb00b25a0e84e73"}
Nov 03 13:13:23.699134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:23.698607    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8" event={"ID":"d5d7e875-ad46-4aa3-8f00-2edf8e609fff","Type":"ContainerStarted","Data":"5233a2ab0c48f1c8ef4326e78a4aec5356cb47066b1a7e193062002fc4f22cf3"}
Nov 03 13:13:23.701949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:23.701928    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x" event={"ID":"6f69df3d-62c1-4209-845b-e7e00cdfdcbb","Type":"ContainerStarted","Data":"1f80244988dde0dac8223444b287f2acc7e7e7901e1766a98f62550b1d5f4e9e"}
Nov 03 13:13:23.704596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:23.704573    4447 generic.go:356] "Generic (PLEG): container finished" podID="eb4af09e-b40a-471e-a82c-96abd2158615" containerID="d30ac6c3c335e8f46c42ceee7856c2c551c28140c7cb735f175a72ac5865aa80" exitCode=0
Nov 03 13:13:23.704670 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:23.704605    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn" event={"ID":"eb4af09e-b40a-471e-a82c-96abd2158615","Type":"ContainerDied","Data":"d30ac6c3c335e8f46c42ceee7856c2c551c28140c7cb735f175a72ac5865aa80"}
Nov 03 13:13:23.769050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:23.768976    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8" podStartSLOduration=3.569537161 podStartE2EDuration="22.768943634s" totalImagesPullingTime="19.199406473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:23.76799591 +0000 UTC m=+291.579664108" watchObservedRunningTime="2026-11-03 13:13:23.768943634 +0000 UTC m=+291.580629596"
Nov 03 13:13:24.710546 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.710512    4447 generic.go:356] "Generic (PLEG): container finished" podID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerID="5233a2ab0c48f1c8ef4326e78a4aec5356cb47066b1a7e193062002fc4f22cf3" exitCode=0
Nov 03 13:13:24.710945 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.710595    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8" event={"ID":"d5d7e875-ad46-4aa3-8f00-2edf8e609fff","Type":"ContainerDied","Data":"5233a2ab0c48f1c8ef4326e78a4aec5356cb47066b1a7e193062002fc4f22cf3"}
Nov 03 13:13:24.712660 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.712638    4447 generic.go:356] "Generic (PLEG): container finished" podID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerID="1f80244988dde0dac8223444b287f2acc7e7e7901e1766a98f62550b1d5f4e9e" exitCode=0
Nov 03 13:13:24.712762 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.712663    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x" event={"ID":"6f69df3d-62c1-4209-845b-e7e00cdfdcbb","Type":"ContainerDied","Data":"1f80244988dde0dac8223444b287f2acc7e7e7901e1766a98f62550b1d5f4e9e"}
Nov 03 13:13:24.843979 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.843952    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:24.895608 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.895581    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:24.927497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.927475    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:24.930707 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.930685    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:24.951112 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.951089    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-util\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-util\") pod \"eb4af09e-b40a-471e-a82c-96abd2158615\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") "
Nov 03 13:13:24.951218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.951144    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-bundle\") pod \"eb4af09e-b40a-471e-a82c-96abd2158615\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") "
Nov 03 13:13:24.951218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.951181    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4af09e-b40a-471e-a82c-96abd2158615-kube-api-access-jqrgr\" (UniqueName: \"kubernetes.io/projected/eb4af09e-b40a-471e-a82c-96abd2158615-kube-api-access-jqrgr\") pod \"eb4af09e-b40a-471e-a82c-96abd2158615\" (UID: \"eb4af09e-b40a-471e-a82c-96abd2158615\") "
Nov 03 13:13:24.951824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.951795    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-bundle" pod "eb4af09e-b40a-471e-a82c-96abd2158615" (UID: "eb4af09e-b40a-471e-a82c-96abd2158615"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:24.953295 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.953274    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4af09e-b40a-471e-a82c-96abd2158615-kube-api-access-jqrgr" pod "eb4af09e-b40a-471e-a82c-96abd2158615" (UID: "eb4af09e-b40a-471e-a82c-96abd2158615"). InnerVolumeSpecName "kube-api-access-jqrgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:24.956397 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:24.956376    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-util" pod "eb4af09e-b40a-471e-a82c-96abd2158615" (UID: "eb4af09e-b40a-471e-a82c-96abd2158615"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:25.052023 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.051933    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-util\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-util\") pod \"1d854b63-1e8f-438b-a86c-08d5eada9b87\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") "
Nov 03 13:13:25.052023 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.051998    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-util\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-util\") pod \"c2cebb8b-2fd3-47f0-8e81-342094772686\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") "
Nov 03 13:13:25.052023 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052022    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d854b63-1e8f-438b-a86c-08d5eada9b87-kube-api-access-7x5mb\" (UniqueName: \"kubernetes.io/projected/1d854b63-1e8f-438b-a86c-08d5eada9b87-kube-api-access-7x5mb\") pod \"1d854b63-1e8f-438b-a86c-08d5eada9b87\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") "
Nov 03 13:13:25.052288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052055    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-util\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-util\") pod \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") "
Nov 03 13:13:25.052288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052078    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-bundle\") pod \"1d854b63-1e8f-438b-a86c-08d5eada9b87\" (UID: \"1d854b63-1e8f-438b-a86c-08d5eada9b87\") "
Nov 03 13:13:25.052288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052101    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2cebb8b-2fd3-47f0-8e81-342094772686-kube-api-access-dhs94\" (UniqueName: \"kubernetes.io/projected/c2cebb8b-2fd3-47f0-8e81-342094772686-kube-api-access-dhs94\") pod \"c2cebb8b-2fd3-47f0-8e81-342094772686\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") "
Nov 03 13:13:25.052288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052162    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-bundle\") pod \"c2cebb8b-2fd3-47f0-8e81-342094772686\" (UID: \"c2cebb8b-2fd3-47f0-8e81-342094772686\") "
Nov 03 13:13:25.052288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052194    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-bundle\") pod \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") "
Nov 03 13:13:25.052288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052241    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-kube-api-access-qdfvd\" (UniqueName: \"kubernetes.io/projected/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-kube-api-access-qdfvd\") pod \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\" (UID: \"6f69df3d-62c1-4209-845b-e7e00cdfdcbb\") "
Nov 03 13:13:25.052629 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052542    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-util\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.052629 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052561    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb4af09e-b40a-471e-a82c-96abd2158615-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.052629 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.052576    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqrgr\" (UniqueName: \"kubernetes.io/projected/eb4af09e-b40a-471e-a82c-96abd2158615-kube-api-access-jqrgr\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.053207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.053145    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-bundle" pod "1d854b63-1e8f-438b-a86c-08d5eada9b87" (UID: "1d854b63-1e8f-438b-a86c-08d5eada9b87"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:25.053207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.053161    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-bundle" pod "6f69df3d-62c1-4209-845b-e7e00cdfdcbb" (UID: "6f69df3d-62c1-4209-845b-e7e00cdfdcbb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:25.053400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.053327    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-bundle" pod "c2cebb8b-2fd3-47f0-8e81-342094772686" (UID: "c2cebb8b-2fd3-47f0-8e81-342094772686"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:25.055060 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.055035    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2cebb8b-2fd3-47f0-8e81-342094772686-kube-api-access-dhs94" pod "c2cebb8b-2fd3-47f0-8e81-342094772686" (UID: "c2cebb8b-2fd3-47f0-8e81-342094772686"). InnerVolumeSpecName "kube-api-access-dhs94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:25.055171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.055035    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d854b63-1e8f-438b-a86c-08d5eada9b87-kube-api-access-7x5mb" pod "1d854b63-1e8f-438b-a86c-08d5eada9b87" (UID: "1d854b63-1e8f-438b-a86c-08d5eada9b87"). InnerVolumeSpecName "kube-api-access-7x5mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:25.055446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.055430    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-kube-api-access-qdfvd" pod "6f69df3d-62c1-4209-845b-e7e00cdfdcbb" (UID: "6f69df3d-62c1-4209-845b-e7e00cdfdcbb"). InnerVolumeSpecName "kube-api-access-qdfvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:25.056876 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.056854    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-util" pod "6f69df3d-62c1-4209-845b-e7e00cdfdcbb" (UID: "6f69df3d-62c1-4209-845b-e7e00cdfdcbb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:25.059958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.059933    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-util" pod "c2cebb8b-2fd3-47f0-8e81-342094772686" (UID: "c2cebb8b-2fd3-47f0-8e81-342094772686"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:25.060014 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.059968    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-util" pod "1d854b63-1e8f-438b-a86c-08d5eada9b87" (UID: "1d854b63-1e8f-438b-a86c-08d5eada9b87"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:25.153014 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.152979    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-util\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.153014 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.153010    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.153014 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.153020    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhs94\" (UniqueName: \"kubernetes.io/projected/c2cebb8b-2fd3-47f0-8e81-342094772686-kube-api-access-dhs94\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.153235 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.153039    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.153235 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.153048    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.153235 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.153056    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfvd\" (UniqueName: \"kubernetes.io/projected/6f69df3d-62c1-4209-845b-e7e00cdfdcbb-kube-api-access-qdfvd\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.153235 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.153064    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d854b63-1e8f-438b-a86c-08d5eada9b87-util\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.153235 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.153072    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2cebb8b-2fd3-47f0-8e81-342094772686-util\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.153235 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.153079    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x5mb\" (UniqueName: \"kubernetes.io/projected/1d854b63-1e8f-438b-a86c-08d5eada9b87-kube-api-access-7x5mb\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:25.718887 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.718861    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x"
Nov 03 13:13:25.718887 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.718867    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxk65x" event={"ID":"6f69df3d-62c1-4209-845b-e7e00cdfdcbb","Type":"ContainerDied","Data":"5d3d5eae7841d6dd110a3a2c5a58890908740e10dfbc1d258c4f401f9555e580"}
Nov 03 13:13:25.719239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.718907    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d3d5eae7841d6dd110a3a2c5a58890908740e10dfbc1d258c4f401f9555e580"
Nov 03 13:13:25.721105 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.721078    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn" event={"ID":"eb4af09e-b40a-471e-a82c-96abd2158615","Type":"ContainerDied","Data":"af22573730e22c6f0a294ab3f6e6022fbf18e11a53d6595960a95c4e29c97c99"}
Nov 03 13:13:25.721212 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.721126    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af22573730e22c6f0a294ab3f6e6022fbf18e11a53d6595960a95c4e29c97c99"
Nov 03 13:13:25.721212 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.721092    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7175mmn"
Nov 03 13:13:25.722781 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.722759    4447 generic.go:356] "Generic (PLEG): container finished" podID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerID="3f2ae4cf538c4a50c21f70dfdd702de02894a11db24895db8332b6ff0290ffdf" exitCode=0
Nov 03 13:13:25.722872 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.722840    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l" event={"ID":"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5","Type":"ContainerDied","Data":"3f2ae4cf538c4a50c21f70dfdd702de02894a11db24895db8332b6ff0290ffdf"}
Nov 03 13:13:25.725077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.725055    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh" event={"ID":"1d854b63-1e8f-438b-a86c-08d5eada9b87","Type":"ContainerDied","Data":"25d6ed898b49b3a911bf4dea93758bc3043ae312c2e491f3940452a0a19913a2"}
Nov 03 13:13:25.725173 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.725078    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpzprh"
Nov 03 13:13:25.725173 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.725089    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25d6ed898b49b3a911bf4dea93758bc3043ae312c2e491f3940452a0a19913a2"
Nov 03 13:13:25.727383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.727365    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv"
Nov 03 13:13:25.727497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.727380    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am9pnv" event={"ID":"c2cebb8b-2fd3-47f0-8e81-342094772686","Type":"ContainerDied","Data":"921b37c38aceb811d3ba7c01092d3d80a18363a56a985b2b66c9cc82d2785abb"}
Nov 03 13:13:25.727497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.727418    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="921b37c38aceb811d3ba7c01092d3d80a18363a56a985b2b66c9cc82d2785abb"
Nov 03 13:13:25.852723 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.852700    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:25.959503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.959471    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-util\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-util\") pod \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") "
Nov 03 13:13:25.959701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.959547    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-kube-api-access-5v8vr\" (UniqueName: \"kubernetes.io/projected/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-kube-api-access-5v8vr\") pod \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") "
Nov 03 13:13:25.959701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.959572    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-bundle\") pod \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\" (UID: \"d5d7e875-ad46-4aa3-8f00-2edf8e609fff\") "
Nov 03 13:13:25.960208 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.960178    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-bundle" pod "d5d7e875-ad46-4aa3-8f00-2edf8e609fff" (UID: "d5d7e875-ad46-4aa3-8f00-2edf8e609fff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:25.961611 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.961585    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-kube-api-access-5v8vr" pod "d5d7e875-ad46-4aa3-8f00-2edf8e609fff" (UID: "d5d7e875-ad46-4aa3-8f00-2edf8e609fff"). InnerVolumeSpecName "kube-api-access-5v8vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:25.963796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:25.963776    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-util" pod "d5d7e875-ad46-4aa3-8f00-2edf8e609fff" (UID: "d5d7e875-ad46-4aa3-8f00-2edf8e609fff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:26.060505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:26.060470    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-util\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:26.060505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:26.060502    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8vr\" (UniqueName: \"kubernetes.io/projected/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-kube-api-access-5v8vr\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:26.060505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:26.060514    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d7e875-ad46-4aa3-8f00-2edf8e609fff-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:26.734084 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:26.734050    4447 generic.go:356] "Generic (PLEG): container finished" podID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerID="24be5212a61c4e74c20c4a83bb4224e2c6f503b97b7c64149ac4757f42ce4565" exitCode=0
Nov 03 13:13:26.734510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:26.734132    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l" event={"ID":"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5","Type":"ContainerDied","Data":"24be5212a61c4e74c20c4a83bb4224e2c6f503b97b7c64149ac4757f42ce4565"}
Nov 03 13:13:26.735889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:26.735868    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8"
Nov 03 13:13:26.735889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:26.735877    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pmd8" event={"ID":"d5d7e875-ad46-4aa3-8f00-2edf8e609fff","Type":"ContainerDied","Data":"326fa2dee1742aaca7783b64ab38fb52efca2d6ecc128fb4a4b1d0aae458b9ee"}
Nov 03 13:13:26.736038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:26.735908    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="326fa2dee1742aaca7783b64ab38fb52efca2d6ecc128fb4a4b1d0aae458b9ee"
Nov 03 13:13:27.866104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:27.866080    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:27.974658 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:27.974624    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-util\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-util\") pod \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") "
Nov 03 13:13:27.974832 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:27.974691    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-bundle\") pod \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") "
Nov 03 13:13:27.974832 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:27.974730    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-kube-api-access-kjnql\" (UniqueName: \"kubernetes.io/projected/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-kube-api-access-kjnql\") pod \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\" (UID: \"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5\") "
Nov 03 13:13:27.975238 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:27.975208    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-bundle" pod "12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" (UID: "12cff7a0-c174-4365-8aaa-ba3fbdf81ce5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:27.977032 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:27.976999    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-kube-api-access-kjnql" pod "12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" (UID: "12cff7a0-c174-4365-8aaa-ba3fbdf81ce5"). InnerVolumeSpecName "kube-api-access-kjnql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:27.982042 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:27.982014    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-util" pod "12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" (UID: "12cff7a0-c174-4365-8aaa-ba3fbdf81ce5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:28.076279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:28.076188    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-bundle\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:28.076279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:28.076216    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjnql\" (UniqueName: \"kubernetes.io/projected/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-kube-api-access-kjnql\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:28.076279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:28.076227    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12cff7a0-c174-4365-8aaa-ba3fbdf81ce5-util\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:28.747221 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:28.747178    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l"
Nov 03 13:13:28.747416 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:28.747176    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ffj4l" event={"ID":"12cff7a0-c174-4365-8aaa-ba3fbdf81ce5","Type":"ContainerDied","Data":"4c498c82e05af08d6c9c53aa40caf66819cfa91ad6130571683fd36ea0e61990"}
Nov 03 13:13:28.747416 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:28.747306    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c498c82e05af08d6c9c53aa40caf66819cfa91ad6130571683fd36ea0e61990"
Nov 03 13:13:31.783544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.783508    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"]
Nov 03 13:13:31.783994 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.783976    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerName="extract"
Nov 03 13:13:31.784038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.783995    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerName="extract"
Nov 03 13:13:31.784038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784007    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerName="util"
Nov 03 13:13:31.784038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784012    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerName="util"
Nov 03 13:13:31.784038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784017    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerName="util"
Nov 03 13:13:31.784038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784022    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerName="util"
Nov 03 13:13:31.784038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784028    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerName="pull"
Nov 03 13:13:31.784038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784032    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerName="pull"
Nov 03 13:13:31.784038 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784040    4447 cpu_manager.go:414] "containerMap: removing container" podUID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784045    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784067    4447 cpu_manager.go:414] "containerMap: removing container" podUID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerName="util"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784072    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerName="util"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784078    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784083    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784088    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784092    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784098    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerName="util"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784102    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerName="util"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784111    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784116    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784125    4447 cpu_manager.go:414] "containerMap: removing container" podUID="eb4af09e-b40a-471e-a82c-96abd2158615" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784130    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4af09e-b40a-471e-a82c-96abd2158615" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784136    4447 cpu_manager.go:414] "containerMap: removing container" podUID="eb4af09e-b40a-471e-a82c-96abd2158615" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784140    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4af09e-b40a-471e-a82c-96abd2158615" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784144    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784148    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784163    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerName="util"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784168    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerName="util"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784175    4447 cpu_manager.go:414] "containerMap: removing container" podUID="eb4af09e-b40a-471e-a82c-96abd2158615" containerName="util"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784180    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4af09e-b40a-471e-a82c-96abd2158615" containerName="util"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784185    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784189    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784196    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784200    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerName="pull"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784207    4447 cpu_manager.go:414] "containerMap: removing container" podUID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784211    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerName="extract"
Nov 03 13:13:31.784258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784267    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d854b63-1e8f-438b-a86c-08d5eada9b87" containerName="extract"
Nov 03 13:13:31.785020 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784277    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d7e875-ad46-4aa3-8f00-2edf8e609fff" containerName="extract"
Nov 03 13:13:31.785020 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784282    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f69df3d-62c1-4209-845b-e7e00cdfdcbb" containerName="extract"
Nov 03 13:13:31.785020 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784289    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2cebb8b-2fd3-47f0-8e81-342094772686" containerName="extract"
Nov 03 13:13:31.785020 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784294    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4af09e-b40a-471e-a82c-96abd2158615" containerName="extract"
Nov 03 13:13:31.785020 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.784301    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cff7a0-c174-4365-8aaa-ba3fbdf81ce5" containerName="extract"
Nov 03 13:13:31.806833 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.806805    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.809327 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.809288    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-manifests\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.809494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.809394    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.809494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.809440    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.809494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.809473    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-tmp\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.809625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.809497    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s425z\" (UniqueName: \"kubernetes.io/projected/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-kube-api-access-s425z\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.809625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.809597    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.817066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.817044    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"]
Nov 03 13:13:31.873453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.873421    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 13:13:31.873621 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.873488    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 13:13:31.876360 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.876324    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 13:13:31.876584 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.876558    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 13:13:31.876584 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.876573    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dvqfb\""
Nov 03 13:13:31.876712 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.876694    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 13:13:31.910991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.910957    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.911130 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.911005    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-manifests\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.911130 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.911035    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.911130 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.911075    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.911130 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.911116    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-tmp\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.911383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.911144    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s425z\" (UniqueName: \"kubernetes.io/projected/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-kube-api-access-s425z\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.911517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.911490    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-manifests\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.913472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.913444    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-tmp\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.913784 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.913765    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.913845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.913801    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.913845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.913806    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:31.919847 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:31.919825    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s425z\" (UniqueName: \"kubernetes.io/projected/bbcaa0fa-ebf0-4311-8edd-73a72b7aed24-kube-api-access-s425z\") pod \"opendatahub-operator-controller-manager-5965948766-cg76t\" (UID: \"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:32.163806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:32.163726    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:32.476903 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:32.476872    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"]
Nov 03 13:13:32.480086 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:13:32.480058    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcaa0fa_ebf0_4311_8edd_73a72b7aed24.slice/crio-005b82108fa30da1f899cde24949a6e2fc5dfa548a8ec747e267527827bf425d WatchSource:0}: Error finding container 005b82108fa30da1f899cde24949a6e2fc5dfa548a8ec747e267527827bf425d: Status 404 returned error can't find the container with id 005b82108fa30da1f899cde24949a6e2fc5dfa548a8ec747e267527827bf425d
Nov 03 13:13:32.766044 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:32.765959    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t" event={"ID":"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24","Type":"ContainerStarted","Data":"005b82108fa30da1f899cde24949a6e2fc5dfa548a8ec747e267527827bf425d"}
Nov 03 13:13:33.419376 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:13:33.419319    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:13:35.783202 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:35.783169    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t" event={"ID":"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24","Type":"ContainerStarted","Data":"495a437bcfa55c28d46689b84d71210af271cac5442e5dbfa7059efe7e412332"}
Nov 03 13:13:36.789395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:36.789363    4447 generic.go:356] "Generic (PLEG): container finished" podID="bbcaa0fa-ebf0-4311-8edd-73a72b7aed24" containerID="495a437bcfa55c28d46689b84d71210af271cac5442e5dbfa7059efe7e412332" exitCode=0
Nov 03 13:13:36.789793 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:36.789405    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t" event={"ID":"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24","Type":"ContainerDied","Data":"495a437bcfa55c28d46689b84d71210af271cac5442e5dbfa7059efe7e412332"}
Nov 03 13:13:36.789971 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:36.789955    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:13:36.990421 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:36.990388    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"]
Nov 03 13:13:36.992716 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:36.992700    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"
Nov 03 13:13:36.995851 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:36.995829    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-l5m2r\""
Nov 03 13:13:37.006108 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.006082    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"]
Nov 03 13:13:37.059600 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.059515    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2007362f-48c3-4563-9159-93e2661244b3-tmp\") pod \"jobset-operator-674cd59dc5-24xgn\" (UID: \"2007362f-48c3-4563-9159-93e2661244b3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"
Nov 03 13:13:37.059600 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.059555    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntrqs\" (UniqueName: \"kubernetes.io/projected/2007362f-48c3-4563-9159-93e2661244b3-kube-api-access-ntrqs\") pod \"jobset-operator-674cd59dc5-24xgn\" (UID: \"2007362f-48c3-4563-9159-93e2661244b3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"
Nov 03 13:13:37.075766 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.075736    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:13:37.075905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.075770    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 13:13:37.160812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.160781    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2007362f-48c3-4563-9159-93e2661244b3-tmp\") pod \"jobset-operator-674cd59dc5-24xgn\" (UID: \"2007362f-48c3-4563-9159-93e2661244b3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"
Nov 03 13:13:37.160812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.160816    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntrqs\" (UniqueName: \"kubernetes.io/projected/2007362f-48c3-4563-9159-93e2661244b3-kube-api-access-ntrqs\") pod \"jobset-operator-674cd59dc5-24xgn\" (UID: \"2007362f-48c3-4563-9159-93e2661244b3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"
Nov 03 13:13:37.161240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.161217    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2007362f-48c3-4563-9159-93e2661244b3-tmp\") pod \"jobset-operator-674cd59dc5-24xgn\" (UID: \"2007362f-48c3-4563-9159-93e2661244b3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"
Nov 03 13:13:37.168306 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.168284    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntrqs\" (UniqueName: \"kubernetes.io/projected/2007362f-48c3-4563-9159-93e2661244b3-kube-api-access-ntrqs\") pod \"jobset-operator-674cd59dc5-24xgn\" (UID: \"2007362f-48c3-4563-9159-93e2661244b3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"
Nov 03 13:13:37.315588 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.315507    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"
Nov 03 13:13:37.625004 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.624982    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn"]
Nov 03 13:13:37.627686 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:13:37.627658    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2007362f_48c3_4563_9159_93e2661244b3.slice/crio-b521e623a450e680e765c2b2b76bf0ac261ba4ccb3705a27870654957cda5262 WatchSource:0}: Error finding container b521e623a450e680e765c2b2b76bf0ac261ba4ccb3705a27870654957cda5262: Status 404 returned error can't find the container with id b521e623a450e680e765c2b2b76bf0ac261ba4ccb3705a27870654957cda5262
Nov 03 13:13:37.795113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.795072    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t" event={"ID":"bbcaa0fa-ebf0-4311-8edd-73a72b7aed24","Type":"ContainerStarted","Data":"5a77522d7634838c2492fd2cb7e095f5679c24a77a56fd2c90d92c8773748090"}
Nov 03 13:13:37.795590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.795215    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:37.796435 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.796413    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn" event={"ID":"2007362f-48c3-4563-9159-93e2661244b3","Type":"ContainerStarted","Data":"b521e623a450e680e765c2b2b76bf0ac261ba4ccb3705a27870654957cda5262"}
Nov 03 13:13:37.832230 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:37.832175    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t" podStartSLOduration=3.0257169 podStartE2EDuration="6.832158202s" totalImagesPullingTime="3.348748902s" totalInitContainerRuntime="457.6924ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:37.831905068 +0000 UTC m=+305.643573302" watchObservedRunningTime="2026-11-03 13:13:37.832158202 +0000 UTC m=+305.643826391"
Nov 03 13:13:40.814542 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:40.814505    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn" event={"ID":"2007362f-48c3-4563-9159-93e2661244b3","Type":"ContainerStarted","Data":"d43058449b63ea2aa7454099b0ce752d8642fab356b7407cca6718c100cbc5c6"}
Nov 03 13:13:40.832132 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:40.832029    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-24xgn" podStartSLOduration=1.8816238379999999 podStartE2EDuration="4.832013892s" totalImagesPullingTime="2.950390054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:40.83127867 +0000 UTC m=+308.642946857" watchObservedRunningTime="2026-11-03 13:13:40.832013892 +0000 UTC m=+308.643682080"
Nov 03 13:13:48.804583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:48.804501    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-cg76t"
Nov 03 13:13:54.793701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:54.793667    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"]
Nov 03 13:13:54.797116 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:54.797100    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:13:54.803772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:54.803749    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 13:13:54.803998 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:54.803946    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 13:13:54.805869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:54.805846    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"]
Nov 03 13:13:54.856553 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:54.856529    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4dnfb\""
Nov 03 13:13:54.930482 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:54.930447    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2426n\" (UniqueName: \"kubernetes.io/projected/7d081d80-35d1-4ee9-a76c-589329f22b5e-kube-api-access-2426n\") pod \"cert-manager-webhook-5dbbbf75cc-jwwdm\" (UID: \"7d081d80-35d1-4ee9-a76c-589329f22b5e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:13:54.930677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:54.930498    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d081d80-35d1-4ee9-a76c-589329f22b5e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-jwwdm\" (UID: \"7d081d80-35d1-4ee9-a76c-589329f22b5e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:13:55.031796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:55.031755    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2426n\" (UniqueName: \"kubernetes.io/projected/7d081d80-35d1-4ee9-a76c-589329f22b5e-kube-api-access-2426n\") pod \"cert-manager-webhook-5dbbbf75cc-jwwdm\" (UID: \"7d081d80-35d1-4ee9-a76c-589329f22b5e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:13:55.031993 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:55.031806    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d081d80-35d1-4ee9-a76c-589329f22b5e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-jwwdm\" (UID: \"7d081d80-35d1-4ee9-a76c-589329f22b5e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:13:55.044882 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:55.044818    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2426n\" (UniqueName: \"kubernetes.io/projected/7d081d80-35d1-4ee9-a76c-589329f22b5e-kube-api-access-2426n\") pod \"cert-manager-webhook-5dbbbf75cc-jwwdm\" (UID: \"7d081d80-35d1-4ee9-a76c-589329f22b5e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:13:55.047475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:55.047454    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d081d80-35d1-4ee9-a76c-589329f22b5e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-jwwdm\" (UID: \"7d081d80-35d1-4ee9-a76c-589329f22b5e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:13:55.108478 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:55.108440    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:13:55.378768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:55.378743    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"]
Nov 03 13:13:55.381275 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:13:55.381241    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d081d80_35d1_4ee9_a76c_589329f22b5e.slice/crio-4c57f121a4c66e45de04feaed7ba222bfc18dc511e57bf1a08cd979f27b1b9f8 WatchSource:0}: Error finding container 4c57f121a4c66e45de04feaed7ba222bfc18dc511e57bf1a08cd979f27b1b9f8: Status 404 returned error can't find the container with id 4c57f121a4c66e45de04feaed7ba222bfc18dc511e57bf1a08cd979f27b1b9f8
Nov 03 13:13:55.889107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:55.889075    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm" event={"ID":"7d081d80-35d1-4ee9-a76c-589329f22b5e","Type":"ContainerStarted","Data":"4c57f121a4c66e45de04feaed7ba222bfc18dc511e57bf1a08cd979f27b1b9f8"}
Nov 03 13:13:59.913729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:59.913691    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm" event={"ID":"7d081d80-35d1-4ee9-a76c-589329f22b5e","Type":"ContainerStarted","Data":"cb4d7fbb25183a1bd0ba4b04dd0ffd2a240c629fbfe4e19c4f57e83fa0826d48"}
Nov 03 13:13:59.914195 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:59.913789    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:13:59.970895 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:13:59.970843    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm" podStartSLOduration=2.282885479 podStartE2EDuration="5.970829827s" totalImagesPullingTime="3.687944348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:59.968990407 +0000 UTC m=+327.780658606" watchObservedRunningTime="2026-11-03 13:13:59.970829827 +0000 UTC m=+327.782498015"
Nov 03 13:14:01.981518 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:01.981481    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"]
Nov 03 13:14:01.985107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:01.985088    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"
Nov 03 13:14:01.991463 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:01.991122    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-t7bhq\""
Nov 03 13:14:01.996791 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:01.996766    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"]
Nov 03 13:14:02.101670 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.101621    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bec38494-4819-48e3-a267-32850474b03b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-l28zx\" (UID: \"bec38494-4819-48e3-a267-32850474b03b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"
Nov 03 13:14:02.101670 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.101663    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdqw\" (UniqueName: \"kubernetes.io/projected/bec38494-4819-48e3-a267-32850474b03b-kube-api-access-2vdqw\") pod \"cert-manager-cainjector-5c57bbd4fc-l28zx\" (UID: \"bec38494-4819-48e3-a267-32850474b03b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"
Nov 03 13:14:02.202951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.202913    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bec38494-4819-48e3-a267-32850474b03b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-l28zx\" (UID: \"bec38494-4819-48e3-a267-32850474b03b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"
Nov 03 13:14:02.203122 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.202960    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdqw\" (UniqueName: \"kubernetes.io/projected/bec38494-4819-48e3-a267-32850474b03b-kube-api-access-2vdqw\") pod \"cert-manager-cainjector-5c57bbd4fc-l28zx\" (UID: \"bec38494-4819-48e3-a267-32850474b03b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"
Nov 03 13:14:02.210472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.210440    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bec38494-4819-48e3-a267-32850474b03b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-l28zx\" (UID: \"bec38494-4819-48e3-a267-32850474b03b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"
Nov 03 13:14:02.210740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.210718    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdqw\" (UniqueName: \"kubernetes.io/projected/bec38494-4819-48e3-a267-32850474b03b-kube-api-access-2vdqw\") pod \"cert-manager-cainjector-5c57bbd4fc-l28zx\" (UID: \"bec38494-4819-48e3-a267-32850474b03b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"
Nov 03 13:14:02.297613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.297531    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"
Nov 03 13:14:02.577926 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.577903    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx"]
Nov 03 13:14:02.580281 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:14:02.580253    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbec38494_4819_48e3_a267_32850474b03b.slice/crio-579fedd3385ed3e066d769818d51db4a07a36ae27416c7c0e4e31ad03d161191 WatchSource:0}: Error finding container 579fedd3385ed3e066d769818d51db4a07a36ae27416c7c0e4e31ad03d161191: Status 404 returned error can't find the container with id 579fedd3385ed3e066d769818d51db4a07a36ae27416c7c0e4e31ad03d161191
Nov 03 13:14:02.929615 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.929580    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx" event={"ID":"bec38494-4819-48e3-a267-32850474b03b","Type":"ContainerStarted","Data":"b5b709bb092569cebdc78c396e793ab1f1ba79acfec51529c381b767de725b75"}
Nov 03 13:14:02.929776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.929629    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx" event={"ID":"bec38494-4819-48e3-a267-32850474b03b","Type":"ContainerStarted","Data":"579fedd3385ed3e066d769818d51db4a07a36ae27416c7c0e4e31ad03d161191"}
Nov 03 13:14:02.945247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:02.945175    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l28zx" podStartSLOduration=1.945156278 podStartE2EDuration="1.945156278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:02.944410668 +0000 UTC m=+330.756078872" watchObservedRunningTime="2026-11-03 13:14:02.945156278 +0000 UTC m=+330.756824471"
Nov 03 13:14:03.410279 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:14:03.410197    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:14:04.118974 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.118932    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-2hskc"]
Nov 03 13:14:04.123182 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.123156    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-2hskc"
Nov 03 13:14:04.135807 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.135779    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qknqr\""
Nov 03 13:14:04.137809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.137786    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-2hskc"]
Nov 03 13:14:04.224665 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.224630    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwttw\" (UniqueName: \"kubernetes.io/projected/8520cc21-c227-4dd1-b921-4be669d8bb86-kube-api-access-bwttw\") pod \"cert-manager-597b95cb84-2hskc\" (UID: \"8520cc21-c227-4dd1-b921-4be669d8bb86\") " pod="cert-manager/cert-manager-597b95cb84-2hskc"
Nov 03 13:14:04.224838 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.224689    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8520cc21-c227-4dd1-b921-4be669d8bb86-bound-sa-token\") pod \"cert-manager-597b95cb84-2hskc\" (UID: \"8520cc21-c227-4dd1-b921-4be669d8bb86\") " pod="cert-manager/cert-manager-597b95cb84-2hskc"
Nov 03 13:14:04.326074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.326038    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwttw\" (UniqueName: \"kubernetes.io/projected/8520cc21-c227-4dd1-b921-4be669d8bb86-kube-api-access-bwttw\") pod \"cert-manager-597b95cb84-2hskc\" (UID: \"8520cc21-c227-4dd1-b921-4be669d8bb86\") " pod="cert-manager/cert-manager-597b95cb84-2hskc"
Nov 03 13:14:04.326269 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.326083    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8520cc21-c227-4dd1-b921-4be669d8bb86-bound-sa-token\") pod \"cert-manager-597b95cb84-2hskc\" (UID: \"8520cc21-c227-4dd1-b921-4be669d8bb86\") " pod="cert-manager/cert-manager-597b95cb84-2hskc"
Nov 03 13:14:04.333511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.333483    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8520cc21-c227-4dd1-b921-4be669d8bb86-bound-sa-token\") pod \"cert-manager-597b95cb84-2hskc\" (UID: \"8520cc21-c227-4dd1-b921-4be669d8bb86\") " pod="cert-manager/cert-manager-597b95cb84-2hskc"
Nov 03 13:14:04.333920 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.333898    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwttw\" (UniqueName: \"kubernetes.io/projected/8520cc21-c227-4dd1-b921-4be669d8bb86-kube-api-access-bwttw\") pod \"cert-manager-597b95cb84-2hskc\" (UID: \"8520cc21-c227-4dd1-b921-4be669d8bb86\") " pod="cert-manager/cert-manager-597b95cb84-2hskc"
Nov 03 13:14:04.435011 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.434971    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-2hskc"
Nov 03 13:14:04.705480 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.705405    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-2hskc"]
Nov 03 13:14:04.708304 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:14:04.708276    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8520cc21_c227_4dd1_b921_4be669d8bb86.slice/crio-5e5aff538b636f8d7b0c5b04fcc3876df4909f58790d4bf9b094defc0609c47b WatchSource:0}: Error finding container 5e5aff538b636f8d7b0c5b04fcc3876df4909f58790d4bf9b094defc0609c47b: Status 404 returned error can't find the container with id 5e5aff538b636f8d7b0c5b04fcc3876df4909f58790d4bf9b094defc0609c47b
Nov 03 13:14:04.940480 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.940445    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-2hskc" event={"ID":"8520cc21-c227-4dd1-b921-4be669d8bb86","Type":"ContainerStarted","Data":"2778f13c87f9f4f80ed8f7313de1c4c8ff872baba8179e5a10283ac1242a0e2a"}
Nov 03 13:14:04.940656 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.940494    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-2hskc" event={"ID":"8520cc21-c227-4dd1-b921-4be669d8bb86","Type":"ContainerStarted","Data":"5e5aff538b636f8d7b0c5b04fcc3876df4909f58790d4bf9b094defc0609c47b"}
Nov 03 13:14:04.957399 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:04.957272    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-2hskc" podStartSLOduration=0.957256056 podStartE2EDuration="957.256056ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:04.956556292 +0000 UTC m=+332.768224480" watchObservedRunningTime="2026-11-03 13:14:04.957256056 +0000 UTC m=+332.768924247"
Nov 03 13:14:05.920155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:05.920126    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jwwdm"
Nov 03 13:14:08.542815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.542784    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"]
Nov 03 13:14:08.548508 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.548483    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.558417 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.558392    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 13:14:08.558549 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.558491    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 13:14:08.559794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.559767    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 13:14:08.559910 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.559796    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 13:14:08.560218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.560195    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8xtzk\""
Nov 03 13:14:08.560329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.560296    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"]
Nov 03 13:14:08.661077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.661047    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbv8p\" (UniqueName: \"kubernetes.io/projected/b0d73ec9-8ac7-4179-86ed-51c93ddc0017-kube-api-access-bbv8p\") pod \"kube-auth-proxy-84d8c65c8f-ltmbm\" (UID: \"b0d73ec9-8ac7-4179-86ed-51c93ddc0017\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.661234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.661092    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d73ec9-8ac7-4179-86ed-51c93ddc0017-tls-certs\") pod \"kube-auth-proxy-84d8c65c8f-ltmbm\" (UID: \"b0d73ec9-8ac7-4179-86ed-51c93ddc0017\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.661234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.661210    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0d73ec9-8ac7-4179-86ed-51c93ddc0017-tmp\") pod \"kube-auth-proxy-84d8c65c8f-ltmbm\" (UID: \"b0d73ec9-8ac7-4179-86ed-51c93ddc0017\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.762332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.762294    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0d73ec9-8ac7-4179-86ed-51c93ddc0017-tmp\") pod \"kube-auth-proxy-84d8c65c8f-ltmbm\" (UID: \"b0d73ec9-8ac7-4179-86ed-51c93ddc0017\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.762521 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.762475    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbv8p\" (UniqueName: \"kubernetes.io/projected/b0d73ec9-8ac7-4179-86ed-51c93ddc0017-kube-api-access-bbv8p\") pod \"kube-auth-proxy-84d8c65c8f-ltmbm\" (UID: \"b0d73ec9-8ac7-4179-86ed-51c93ddc0017\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.762603 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.762537    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d73ec9-8ac7-4179-86ed-51c93ddc0017-tls-certs\") pod \"kube-auth-proxy-84d8c65c8f-ltmbm\" (UID: \"b0d73ec9-8ac7-4179-86ed-51c93ddc0017\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.765081 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.765053    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0d73ec9-8ac7-4179-86ed-51c93ddc0017-tmp\") pod \"kube-auth-proxy-84d8c65c8f-ltmbm\" (UID: \"b0d73ec9-8ac7-4179-86ed-51c93ddc0017\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.765216 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.765150    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d73ec9-8ac7-4179-86ed-51c93ddc0017-tls-certs\") pod \"kube-auth-proxy-84d8c65c8f-ltmbm\" (UID: \"b0d73ec9-8ac7-4179-86ed-51c93ddc0017\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.773752 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.773726    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbv8p\" (UniqueName: \"kubernetes.io/projected/b0d73ec9-8ac7-4179-86ed-51c93ddc0017-kube-api-access-bbv8p\") pod \"kube-auth-proxy-84d8c65c8f-ltmbm\" (UID: \"b0d73ec9-8ac7-4179-86ed-51c93ddc0017\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:08.860469 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:08.860382    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"
Nov 03 13:14:09.172938 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:09.172854    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm"]
Nov 03 13:14:09.176051 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:14:09.176022    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d73ec9_8ac7_4179_86ed_51c93ddc0017.slice/crio-258f47f917f3063d1cf491a1ec5f67ee47d1a3906fa7bed033dde070b057e0eb WatchSource:0}: Error finding container 258f47f917f3063d1cf491a1ec5f67ee47d1a3906fa7bed033dde070b057e0eb: Status 404 returned error can't find the container with id 258f47f917f3063d1cf491a1ec5f67ee47d1a3906fa7bed033dde070b057e0eb
Nov 03 13:14:09.967078 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:09.967035    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm" event={"ID":"b0d73ec9-8ac7-4179-86ed-51c93ddc0017","Type":"ContainerStarted","Data":"258f47f917f3063d1cf491a1ec5f67ee47d1a3906fa7bed033dde070b057e0eb"}
Nov 03 13:14:10.600222 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.600180    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hk472"]
Nov 03 13:14:10.607036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.607009    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:10.612144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.611911    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 13:14:10.612144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.612018    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 13:14:10.612144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.612122    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 13:14:10.614209 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.614185    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hk472"]
Nov 03 13:14:10.662011 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.661982    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lsg7c\""
Nov 03 13:14:10.701189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.701141    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hk472"]
Nov 03 13:14:10.781858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.781824    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jk8\" (UniqueName: \"kubernetes.io/projected/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-kube-api-access-s5jk8\") pod \"limitador-limitador-74dfcfb894-hk472\" (UID: \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:10.782045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.781880    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-config-file\") pod \"limitador-limitador-74dfcfb894-hk472\" (UID: \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:10.882853 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.882778    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jk8\" (UniqueName: \"kubernetes.io/projected/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-kube-api-access-s5jk8\") pod \"limitador-limitador-74dfcfb894-hk472\" (UID: \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:10.882853 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.882848    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-config-file\") pod \"limitador-limitador-74dfcfb894-hk472\" (UID: \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:10.883597 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.883570    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-config-file\") pod \"limitador-limitador-74dfcfb894-hk472\" (UID: \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:10.894774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.894745    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jk8\" (UniqueName: \"kubernetes.io/projected/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-kube-api-access-s5jk8\") pod \"limitador-limitador-74dfcfb894-hk472\" (UID: \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:10.923268 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:10.923237    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:11.232957 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.232800    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hk472"]
Nov 03 13:14:11.235483 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:14:11.235456    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b7b036_5f1c_45c6_9f8b_cf1024680a6e.slice/crio-9a1694385b90e9352aa1697802cf01d2a783a5ac84b8c147bc2170d77f677af9 WatchSource:0}: Error finding container 9a1694385b90e9352aa1697802cf01d2a783a5ac84b8c147bc2170d77f677af9: Status 404 returned error can't find the container with id 9a1694385b90e9352aa1697802cf01d2a783a5ac84b8c147bc2170d77f677af9
Nov 03 13:14:11.564187 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.564108    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8c92c"]
Nov 03 13:14:11.593025 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.592990    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8c92c"]
Nov 03 13:14:11.593221 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.593086    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8c92c"
Nov 03 13:14:11.604534 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.604509    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kk72q\""
Nov 03 13:14:11.692877 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.692854    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9mt\" (UniqueName: \"kubernetes.io/projected/f9ebf413-32b1-4c7a-94cf-666ae4771f97-kube-api-access-rf9mt\") pod \"authorino-57f7f8659f-8c92c\" (UID: \"f9ebf413-32b1-4c7a-94cf-666ae4771f97\") " pod="kuadrant-system/authorino-57f7f8659f-8c92c"
Nov 03 13:14:11.794360 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.794318    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9mt\" (UniqueName: \"kubernetes.io/projected/f9ebf413-32b1-4c7a-94cf-666ae4771f97-kube-api-access-rf9mt\") pod \"authorino-57f7f8659f-8c92c\" (UID: \"f9ebf413-32b1-4c7a-94cf-666ae4771f97\") " pod="kuadrant-system/authorino-57f7f8659f-8c92c"
Nov 03 13:14:11.806427 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.806400    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"]
Nov 03 13:14:11.809101 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.809084    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:11.811831 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.811809    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9mt\" (UniqueName: \"kubernetes.io/projected/f9ebf413-32b1-4c7a-94cf-666ae4771f97-kube-api-access-rf9mt\") pod \"authorino-57f7f8659f-8c92c\" (UID: \"f9ebf413-32b1-4c7a-94cf-666ae4771f97\") " pod="kuadrant-system/authorino-57f7f8659f-8c92c"
Nov 03 13:14:11.823434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.823414    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 13:14:11.823586 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.823573    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 13:14:11.824263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.824245    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"]
Nov 03 13:14:11.825321 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.825300    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 13:14:11.894954 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.894921    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-tbqmn\""
Nov 03 13:14:11.909187 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.909159    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8c92c"
Nov 03 13:14:11.979772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.979737    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm" event={"ID":"b0d73ec9-8ac7-4179-86ed-51c93ddc0017","Type":"ContainerStarted","Data":"0eb4fb3d39295947301c5c76258693098cd107942f853fefbb67495c02611b8e"}
Nov 03 13:14:11.981129 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.981103    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472" event={"ID":"43b7b036-5f1c-45c6-9f8b-cf1024680a6e","Type":"ContainerStarted","Data":"9a1694385b90e9352aa1697802cf01d2a783a5ac84b8c147bc2170d77f677af9"}
Nov 03 13:14:11.995589 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.995555    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ab4053b-cfb1-49c2-a517-087f645e4dec-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:11.995701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.995604    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ab4053b-cfb1-49c2-a517-087f645e4dec-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:11.995701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.995676    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1ab4053b-cfb1-49c2-a517-087f645e4dec-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:11.995792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.995724    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdkw\" (UniqueName: \"kubernetes.io/projected/1ab4053b-cfb1-49c2-a517-087f645e4dec-kube-api-access-ptdkw\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:11.997244 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:11.997203    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-ltmbm" podStartSLOduration=1.530489063 podStartE2EDuration="3.997187677s" totalImagesPullingTime="2.466698614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:11.996105955 +0000 UTC m=+339.807774161" watchObservedRunningTime="2026-11-03 13:14:11.997187677 +0000 UTC m=+339.808855866"
Nov 03 13:14:12.096890 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.096821    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ab4053b-cfb1-49c2-a517-087f645e4dec-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:12.096890 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.096873    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1ab4053b-cfb1-49c2-a517-087f645e4dec-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:12.097115 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.096909    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdkw\" (UniqueName: \"kubernetes.io/projected/1ab4053b-cfb1-49c2-a517-087f645e4dec-kube-api-access-ptdkw\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:12.097115 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.096972    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ab4053b-cfb1-49c2-a517-087f645e4dec-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:12.097376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.097321    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ab4053b-cfb1-49c2-a517-087f645e4dec-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:12.097420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.097392    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1ab4053b-cfb1-49c2-a517-087f645e4dec-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:12.097802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.097785    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ab4053b-cfb1-49c2-a517-087f645e4dec-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:12.104258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.104233    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdkw\" (UniqueName: \"kubernetes.io/projected/1ab4053b-cfb1-49c2-a517-087f645e4dec-kube-api-access-ptdkw\") pod \"kserve-module-controller-manager-7566cf5fc8-8w2nb\" (UID: \"1ab4053b-cfb1-49c2-a517-087f645e4dec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:12.122295 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.122268    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:12.184037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.183858    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8c92c"]
Nov 03 13:14:12.194296 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:14:12.194262    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9ebf413_32b1_4c7a_94cf_666ae4771f97.slice/crio-3983187c322480dee9ed38308104b26f41a66e707ecc141d79c1ca30f1b42f9a WatchSource:0}: Error finding container 3983187c322480dee9ed38308104b26f41a66e707ecc141d79c1ca30f1b42f9a: Status 404 returned error can't find the container with id 3983187c322480dee9ed38308104b26f41a66e707ecc141d79c1ca30f1b42f9a
Nov 03 13:14:12.406541 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.406456    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"]
Nov 03 13:14:12.987504 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.987468    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8c92c" event={"ID":"f9ebf413-32b1-4c7a-94cf-666ae4771f97","Type":"ContainerStarted","Data":"3983187c322480dee9ed38308104b26f41a66e707ecc141d79c1ca30f1b42f9a"}
Nov 03 13:14:12.988773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:12.988750    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb" event={"ID":"1ab4053b-cfb1-49c2-a517-087f645e4dec","Type":"ContainerStarted","Data":"897de09f9c1e768c1406d6144609afa3a761a7920fe08f0498d539d350290c4b"}
Nov 03 13:14:15.002016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:15.001977    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8c92c" event={"ID":"f9ebf413-32b1-4c7a-94cf-666ae4771f97","Type":"ContainerStarted","Data":"6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990"}
Nov 03 13:14:15.003535 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:15.003510    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472" event={"ID":"43b7b036-5f1c-45c6-9f8b-cf1024680a6e","Type":"ContainerStarted","Data":"56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295"}
Nov 03 13:14:15.003661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:15.003580    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:15.004939 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:15.004916    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb" event={"ID":"1ab4053b-cfb1-49c2-a517-087f645e4dec","Type":"ContainerStarted","Data":"4e3be435097728f175d2e3a0d02846e2e67556ac679350617df3258f26589fbe"}
Nov 03 13:14:15.005064 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:15.005051    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:15.016206 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:15.016166    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-8c92c" podStartSLOduration=1.650468272 podStartE2EDuration="4.016155095s" totalImagesPullingTime="2.365686823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:15.014722949 +0000 UTC m=+342.826391162" watchObservedRunningTime="2026-11-03 13:14:15.016155095 +0000 UTC m=+342.827823283"
Nov 03 13:14:15.036598 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:15.036554    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb" podStartSLOduration=2.129574464 podStartE2EDuration="4.036542207s" totalImagesPullingTime="1.906967743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:15.034943805 +0000 UTC m=+342.846611995" watchObservedRunningTime="2026-11-03 13:14:15.036542207 +0000 UTC m=+342.848210395"
Nov 03 13:14:15.050781 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:15.050736    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472" podStartSLOduration=2.153541653 podStartE2EDuration="5.050723382s" totalImagesPullingTime="2.897181729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:15.049169105 +0000 UTC m=+342.860837294" watchObservedRunningTime="2026-11-03 13:14:15.050723382 +0000 UTC m=+342.862391569"
Nov 03 13:14:15.563412 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:15.563379    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8c92c"]
Nov 03 13:14:17.014999 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:17.014958    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-8c92c" podUID="f9ebf413-32b1-4c7a-94cf-666ae4771f97" containerName="authorino" containerID="cri-o://6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990" gracePeriod=30
Nov 03 13:14:17.256909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:17.256887    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8c92c"
Nov 03 13:14:17.341640 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:17.341556    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ebf413-32b1-4c7a-94cf-666ae4771f97-kube-api-access-rf9mt\" (UniqueName: \"kubernetes.io/projected/f9ebf413-32b1-4c7a-94cf-666ae4771f97-kube-api-access-rf9mt\") pod \"f9ebf413-32b1-4c7a-94cf-666ae4771f97\" (UID: \"f9ebf413-32b1-4c7a-94cf-666ae4771f97\") "
Nov 03 13:14:17.343669 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:17.343645    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ebf413-32b1-4c7a-94cf-666ae4771f97-kube-api-access-rf9mt" pod "f9ebf413-32b1-4c7a-94cf-666ae4771f97" (UID: "f9ebf413-32b1-4c7a-94cf-666ae4771f97"). InnerVolumeSpecName "kube-api-access-rf9mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:17.442550 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:17.442510    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9mt\" (UniqueName: \"kubernetes.io/projected/f9ebf413-32b1-4c7a-94cf-666ae4771f97-kube-api-access-rf9mt\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:18.021063 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.021022    4447 generic.go:356] "Generic (PLEG): container finished" podID="f9ebf413-32b1-4c7a-94cf-666ae4771f97" containerID="6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990" exitCode=0
Nov 03 13:14:18.021492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.021079    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8c92c"
Nov 03 13:14:18.021492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.021083    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8c92c" event={"ID":"f9ebf413-32b1-4c7a-94cf-666ae4771f97","Type":"ContainerDied","Data":"6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990"}
Nov 03 13:14:18.021492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.021198    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8c92c" event={"ID":"f9ebf413-32b1-4c7a-94cf-666ae4771f97","Type":"ContainerDied","Data":"3983187c322480dee9ed38308104b26f41a66e707ecc141d79c1ca30f1b42f9a"}
Nov 03 13:14:18.021492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.021227    4447 scope.go:122] "RemoveContainer" containerID="6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990"
Nov 03 13:14:18.030654 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.030635    4447 scope.go:122] "RemoveContainer" containerID="6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990"
Nov 03 13:14:18.030882 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:14:18.030864    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990\": container with ID starting with 6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990 not found: ID does not exist" containerID="6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990"
Nov 03 13:14:18.030931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.030891    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990"} err="failed to get container status \"6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990\": rpc error: code = NotFound desc = could not find container \"6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990\": container with ID starting with 6219deb0924661e0e47370b3a7b64751eebfcaa606ffea209eb64b26b1064990 not found: ID does not exist"
Nov 03 13:14:18.040883 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.040861    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8c92c"]
Nov 03 13:14:18.043497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.043479    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8c92c"]
Nov 03 13:14:18.317836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:18.317759    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ebf413-32b1-4c7a-94cf-666ae4771f97" path="/var/lib/kubelet/pods/f9ebf413-32b1-4c7a-94cf-666ae4771f97/volumes"
Nov 03 13:14:25.412653 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:25.412617    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hk472"]
Nov 03 13:14:25.413081 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:25.412833    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472" podUID="43b7b036-5f1c-45c6-9f8b-cf1024680a6e" containerName="limitador" containerID="cri-o://56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295" gracePeriod=30
Nov 03 13:14:25.413550 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:25.413496    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:25.956568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:25.956548    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:26.006001 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.005916    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-kube-api-access-s5jk8\" (UniqueName: \"kubernetes.io/projected/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-kube-api-access-s5jk8\") pod \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\" (UID: \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\") "
Nov 03 13:14:26.006001 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.005960    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-config-file\" (UniqueName: \"kubernetes.io/configmap/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-config-file\") pod \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\" (UID: \"43b7b036-5f1c-45c6-9f8b-cf1024680a6e\") "
Nov 03 13:14:26.006394 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.006337    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-config-file" pod "43b7b036-5f1c-45c6-9f8b-cf1024680a6e" (UID: "43b7b036-5f1c-45c6-9f8b-cf1024680a6e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:26.008035 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.008010    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-kube-api-access-s5jk8" pod "43b7b036-5f1c-45c6-9f8b-cf1024680a6e" (UID: "43b7b036-5f1c-45c6-9f8b-cf1024680a6e"). InnerVolumeSpecName "kube-api-access-s5jk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:26.074008 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.073977    4447 generic.go:356] "Generic (PLEG): container finished" podID="43b7b036-5f1c-45c6-9f8b-cf1024680a6e" containerID="56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295" exitCode=0
Nov 03 13:14:26.074193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.074023    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472"
Nov 03 13:14:26.074193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.074067    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472" event={"ID":"43b7b036-5f1c-45c6-9f8b-cf1024680a6e","Type":"ContainerDied","Data":"56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295"}
Nov 03 13:14:26.074193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.074125    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hk472" event={"ID":"43b7b036-5f1c-45c6-9f8b-cf1024680a6e","Type":"ContainerDied","Data":"9a1694385b90e9352aa1697802cf01d2a783a5ac84b8c147bc2170d77f677af9"}
Nov 03 13:14:26.074193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.074153    4447 scope.go:122] "RemoveContainer" containerID="56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295"
Nov 03 13:14:26.083627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.083610    4447 scope.go:122] "RemoveContainer" containerID="56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295"
Nov 03 13:14:26.083890 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:14:26.083872    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295\": container with ID starting with 56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295 not found: ID does not exist" containerID="56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295"
Nov 03 13:14:26.083960 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.083904    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295"} err="failed to get container status \"56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295\": rpc error: code = NotFound desc = could not find container \"56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295\": container with ID starting with 56bd2ec266153dbcf25af5215d9b4f530e3514d8250943d20c5fa4ac65cd7295 not found: ID does not exist"
Nov 03 13:14:26.098537 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.098516    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hk472"]
Nov 03 13:14:26.106715 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.106692    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jk8\" (UniqueName: \"kubernetes.io/projected/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-kube-api-access-s5jk8\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:26.106715 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.106715    4447 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/43b7b036-5f1c-45c6-9f8b-cf1024680a6e-config-file\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:26.108458 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.108437    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hk472"]
Nov 03 13:14:26.318254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:26.318177    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b7b036-5f1c-45c6-9f8b-cf1024680a6e" path="/var/lib/kubelet/pods/43b7b036-5f1c-45c6-9f8b-cf1024680a6e/volumes"
Nov 03 13:14:31.532636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.532589    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"]
Nov 03 13:14:31.533110 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.533045    4447 cpu_manager.go:414] "containerMap: removing container" podUID="43b7b036-5f1c-45c6-9f8b-cf1024680a6e" containerName="limitador"
Nov 03 13:14:31.533110 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.533061    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b7b036-5f1c-45c6-9f8b-cf1024680a6e" containerName="limitador"
Nov 03 13:14:31.533110 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.533107    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f9ebf413-32b1-4c7a-94cf-666ae4771f97" containerName="authorino"
Nov 03 13:14:31.533208 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.533113    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ebf413-32b1-4c7a-94cf-666ae4771f97" containerName="authorino"
Nov 03 13:14:31.533208 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.533183    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ebf413-32b1-4c7a-94cf-666ae4771f97" containerName="authorino"
Nov 03 13:14:31.533208 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.533195    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b7b036-5f1c-45c6-9f8b-cf1024680a6e" containerName="limitador"
Nov 03 13:14:31.537898 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.537882    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:31.544332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.544303    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-z9jjd\""
Nov 03 13:14:31.544541 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.544494    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 13:14:31.544541 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.544539    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:14:31.544936 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.544603    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:14:31.545601 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.545576    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"]
Nov 03 13:14:31.554600 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.554574    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdkl\" (UniqueName: \"kubernetes.io/projected/3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4-kube-api-access-2rdkl\") pod \"postgres-7cdbf6dc44-fjs8c\" (UID: \"3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:31.554727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.554670    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4-data\") pod \"postgres-7cdbf6dc44-fjs8c\" (UID: \"3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:31.655296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.655257    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdkl\" (UniqueName: \"kubernetes.io/projected/3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4-kube-api-access-2rdkl\") pod \"postgres-7cdbf6dc44-fjs8c\" (UID: \"3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:31.655479 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.655317    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4-data\") pod \"postgres-7cdbf6dc44-fjs8c\" (UID: \"3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:31.655652 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.655635    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4-data\") pod \"postgres-7cdbf6dc44-fjs8c\" (UID: \"3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:31.666169 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.666140    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdkl\" (UniqueName: \"kubernetes.io/projected/3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4-kube-api-access-2rdkl\") pod \"postgres-7cdbf6dc44-fjs8c\" (UID: \"3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:31.851455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:31.851376    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:32.122915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:32.122768    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"]
Nov 03 13:14:32.125654 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:14:32.125621    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7ec807_b74e_4dfe_a48d_a1531d7ec2c4.slice/crio-99b39560ca1aeb86ec67f5364720a92d2f0342ca88d53ad45107334d40ccb698 WatchSource:0}: Error finding container 99b39560ca1aeb86ec67f5364720a92d2f0342ca88d53ad45107334d40ccb698: Status 404 returned error can't find the container with id 99b39560ca1aeb86ec67f5364720a92d2f0342ca88d53ad45107334d40ccb698
Nov 03 13:14:33.109783 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:33.109737    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c" event={"ID":"3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4","Type":"ContainerStarted","Data":"99b39560ca1aeb86ec67f5364720a92d2f0342ca88d53ad45107334d40ccb698"}
Nov 03 13:14:33.409568 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:14:33.409473    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:14:36.011486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:36.011455    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8w2nb"
Nov 03 13:14:37.949692 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:37.949665    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 13:14:38.140654 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:38.140620    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c" event={"ID":"3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4","Type":"ContainerStarted","Data":"2f2d10e56c299e3006367dc2c8b5f491c7bd84cbe76f383bdcc8d0a5b0e4f92c"}
Nov 03 13:14:38.140851 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:38.140698    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:44.176745 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:44.176711    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c"
Nov 03 13:14:44.192470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:44.192420    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjs8c" podStartSLOduration=7.372531267 podStartE2EDuration="13.192406325s" totalImagesPullingTime="5.819875058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:38.17033049 +0000 UTC m=+365.981998696" watchObservedRunningTime="2026-11-03 13:14:44.192406325 +0000 UTC m=+372.004074512"
Nov 03 13:14:50.930841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:50.930798    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5776b58b5b-q6zgl"]
Nov 03 13:14:50.934458 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:50.934435    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:50.942603 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:50.942579    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 13:14:50.943370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:50.943330    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qn5lb\""
Nov 03 13:14:50.943483 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:50.943462    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 13:14:50.943549 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:50.943466    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 13:14:50.946053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:50.946034    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5776b58b5b-q6zgl"]
Nov 03 13:14:51.037647 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.037611    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-webhook-cert\") pod \"maas-controller-5776b58b5b-q6zgl\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") " pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.037824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.037778    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsck\" (UniqueName: \"kubernetes.io/projected/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-kube-api-access-jnsck\") pod \"maas-controller-5776b58b5b-q6zgl\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") " pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.037877 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.037859    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-metrics-certs\") pod \"maas-controller-5776b58b5b-q6zgl\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") " pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.139210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.139165    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsck\" (UniqueName: \"kubernetes.io/projected/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-kube-api-access-jnsck\") pod \"maas-controller-5776b58b5b-q6zgl\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") " pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.139365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.139255    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-metrics-certs\") pod \"maas-controller-5776b58b5b-q6zgl\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") " pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.139365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.139309    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-webhook-cert\") pod \"maas-controller-5776b58b5b-q6zgl\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") " pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.141861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.141831    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-metrics-certs\") pod \"maas-controller-5776b58b5b-q6zgl\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") " pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.141972 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.141944    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-webhook-cert\") pod \"maas-controller-5776b58b5b-q6zgl\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") " pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.146660 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.146636    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsck\" (UniqueName: \"kubernetes.io/projected/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-kube-api-access-jnsck\") pod \"maas-controller-5776b58b5b-q6zgl\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") " pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.247226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.247139    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:51.409601 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.409564    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bc865c477-6g9s8"]
Nov 03 13:14:51.412352 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.412327    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.424555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.424523    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc865c477-6g9s8"]
Nov 03 13:14:51.522603 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.522572    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5776b58b5b-q6zgl"]
Nov 03 13:14:51.525015 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:14:51.524987    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67aa6d1_046f_4aa1_8a48_5379c8d8ba05.slice/crio-6ef7a330bf86a49dc071df530681e4a0cb8e823ec5ad181db3cb166653baa83c WatchSource:0}: Error finding container 6ef7a330bf86a49dc071df530681e4a0cb8e823ec5ad181db3cb166653baa83c: Status 404 returned error can't find the container with id 6ef7a330bf86a49dc071df530681e4a0cb8e823ec5ad181db3cb166653baa83c
Nov 03 13:14:51.542988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.542943    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-webhook-cert\") pod \"maas-controller-bc865c477-6g9s8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") " pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.543122 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.543002    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2hbq\" (UniqueName: \"kubernetes.io/projected/72d09d90-d90c-4afe-bd2f-c95980f250e8-kube-api-access-r2hbq\") pod \"maas-controller-bc865c477-6g9s8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") " pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.543192 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.543167    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-metrics-certs\") pod \"maas-controller-bc865c477-6g9s8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") " pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.644614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.644580    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-webhook-cert\") pod \"maas-controller-bc865c477-6g9s8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") " pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.644614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.644616    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2hbq\" (UniqueName: \"kubernetes.io/projected/72d09d90-d90c-4afe-bd2f-c95980f250e8-kube-api-access-r2hbq\") pod \"maas-controller-bc865c477-6g9s8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") " pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.644885 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.644679    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-metrics-certs\") pod \"maas-controller-bc865c477-6g9s8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") " pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.647590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.647558    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-metrics-certs\") pod \"maas-controller-bc865c477-6g9s8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") " pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.647771 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.647672    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-webhook-cert\") pod \"maas-controller-bc865c477-6g9s8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") " pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.655125 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.655097    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2hbq\" (UniqueName: \"kubernetes.io/projected/72d09d90-d90c-4afe-bd2f-c95980f250e8-kube-api-access-r2hbq\") pod \"maas-controller-bc865c477-6g9s8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") " pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.724735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.724697    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:51.996678 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:51.996652    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc865c477-6g9s8"]
Nov 03 13:14:51.999234 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:14:51.999203    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d09d90_d90c_4afe_bd2f_c95980f250e8.slice/crio-1e10d62de8e21d2db3ad46d726f7e54a28eb870c7237069f527363ba65c11ae9 WatchSource:0}: Error finding container 1e10d62de8e21d2db3ad46d726f7e54a28eb870c7237069f527363ba65c11ae9: Status 404 returned error can't find the container with id 1e10d62de8e21d2db3ad46d726f7e54a28eb870c7237069f527363ba65c11ae9
Nov 03 13:14:52.212793 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:52.212759    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-6g9s8" event={"ID":"72d09d90-d90c-4afe-bd2f-c95980f250e8","Type":"ContainerStarted","Data":"1e10d62de8e21d2db3ad46d726f7e54a28eb870c7237069f527363ba65c11ae9"}
Nov 03 13:14:52.213904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:52.213873    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5776b58b5b-q6zgl" event={"ID":"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05","Type":"ContainerStarted","Data":"6ef7a330bf86a49dc071df530681e4a0cb8e823ec5ad181db3cb166653baa83c"}
Nov 03 13:14:56.239932 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.239896    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-6g9s8" event={"ID":"72d09d90-d90c-4afe-bd2f-c95980f250e8","Type":"ContainerStarted","Data":"0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67"}
Nov 03 13:14:56.240524 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.240504    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:14:56.241813 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.241786    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5776b58b5b-q6zgl" event={"ID":"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05","Type":"ContainerStarted","Data":"a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272"}
Nov 03 13:14:56.241917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.241848    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:14:56.261051 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.260991    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bc865c477-6g9s8" podStartSLOduration=1.596479515 podStartE2EDuration="5.260979605s" totalImagesPullingTime="3.66450009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:56.2591542 +0000 UTC m=+384.070822409" watchObservedRunningTime="2026-11-03 13:14:56.260979605 +0000 UTC m=+384.072647792"
Nov 03 13:14:56.279875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.279825    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5776b58b5b-q6zgl" podStartSLOduration=2.131252677 podStartE2EDuration="6.279811593s" totalImagesPullingTime="4.148558916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:56.276813007 +0000 UTC m=+384.088481238" watchObservedRunningTime="2026-11-03 13:14:56.279811593 +0000 UTC m=+384.091479777"
Nov 03 13:14:56.842099 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.842062    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"]
Nov 03 13:14:56.846141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.846108    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:56.851601 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.851576    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 13:14:56.855033 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.854971    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"]
Nov 03 13:14:56.862686 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.862438    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-wx5zz\""
Nov 03 13:14:56.862795 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.862685    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 13:14:56.891618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:56.891583    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 13:14:57.004227 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.004185    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ddba5e-73fe-472c-bff2-e98dafa5977a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.004426 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.004239    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2wf7\" (UniqueName: \"kubernetes.io/projected/63ddba5e-73fe-472c-bff2-e98dafa5977a-kube-api-access-x2wf7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.004426 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.004362    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/63ddba5e-73fe-472c-bff2-e98dafa5977a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.004426 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.004418    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ddba5e-73fe-472c-bff2-e98dafa5977a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.105276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.105188    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/63ddba5e-73fe-472c-bff2-e98dafa5977a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.105276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.105245    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ddba5e-73fe-472c-bff2-e98dafa5977a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.105497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.105282    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ddba5e-73fe-472c-bff2-e98dafa5977a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.105497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.105317    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2wf7\" (UniqueName: \"kubernetes.io/projected/63ddba5e-73fe-472c-bff2-e98dafa5977a-kube-api-access-x2wf7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.105976 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.105950    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ddba5e-73fe-472c-bff2-e98dafa5977a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.106228 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.106205    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/63ddba5e-73fe-472c-bff2-e98dafa5977a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.107753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.107731    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ddba5e-73fe-472c-bff2-e98dafa5977a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.112700 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.112679    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2wf7\" (UniqueName: \"kubernetes.io/projected/63ddba5e-73fe-472c-bff2-e98dafa5977a-kube-api-access-x2wf7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2trjw\" (UID: \"63ddba5e-73fe-472c-bff2-e98dafa5977a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.168699 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.168666    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:14:57.441809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:57.441752    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"]
Nov 03 13:14:57.444296 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:14:57.444269    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ddba5e_73fe_472c_bff2_e98dafa5977a.slice/crio-5816ef75e3eb5e820f9e3c01d4f5d08ca9254f36d13de9962cc803a095f4d20d WatchSource:0}: Error finding container 5816ef75e3eb5e820f9e3c01d4f5d08ca9254f36d13de9962cc803a095f4d20d: Status 404 returned error can't find the container with id 5816ef75e3eb5e820f9e3c01d4f5d08ca9254f36d13de9962cc803a095f4d20d
Nov 03 13:14:58.254942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:14:58.254908    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw" event={"ID":"63ddba5e-73fe-472c-bff2-e98dafa5977a","Type":"ContainerStarted","Data":"5816ef75e3eb5e820f9e3c01d4f5d08ca9254f36d13de9962cc803a095f4d20d"}
Nov 03 13:15:03.418554 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:15:03.418511    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:15:05.313030 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.301154    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-tbkh6"]
Nov 03 13:15:05.345595 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.345560    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-tbkh6"]
Nov 03 13:15:05.345998 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.345612    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"]
Nov 03 13:15:05.348941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.348919    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:05.349567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.349549    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"]
Nov 03 13:15:05.349658 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.349648    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:05.360596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.356193    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6zhqm"]
Nov 03 13:15:05.377377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.369512    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:05.377377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.376070    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 13:15:05.377377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.376154    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 13:15:05.377377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.376593    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pkjnj\""
Nov 03 13:15:05.377377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.376705    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-brr74\""
Nov 03 13:15:05.377899 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.377886    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6zhqm"]
Nov 03 13:15:05.382376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.382334    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-945dt"]
Nov 03 13:15:05.389494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.387187    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rffm6\""
Nov 03 13:15:05.389494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.389003    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 13:15:05.389494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.389478    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:05.392238 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.392217    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-945dt"]
Nov 03 13:15:05.430773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.424938    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37bc192f-69cc-4825-9a15-48bb8a6aae51-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-jbr2p\" (UID: \"37bc192f-69cc-4825-9a15-48bb8a6aae51\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:05.430773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.424983    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bggvb\" (UniqueName: \"kubernetes.io/projected/9a2d78c7-0ebe-45dd-9683-02279bbfff20-kube-api-access-bggvb\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:05.430773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.425026    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a2d78c7-0ebe-45dd-9683-02279bbfff20-cert\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:05.430773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.425063    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8v7f\" (UniqueName: \"kubernetes.io/projected/d5388bcf-9195-4a03-89b5-d50dcb20b89d-kube-api-access-j8v7f\") pod \"kserve-controller-manager-776cc66d47-tbkh6\" (UID: \"d5388bcf-9195-4a03-89b5-d50dcb20b89d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:05.430773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.425094    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n7th\" (UniqueName: \"kubernetes.io/projected/e77dd41f-1fc2-4b1e-be96-902b2186b80e-kube-api-access-9n7th\") pod \"model-serving-api-75bd89fdfc-6zhqm\" (UID: \"e77dd41f-1fc2-4b1e-be96-902b2186b80e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:05.430773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.425132    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2d78c7-0ebe-45dd-9683-02279bbfff20-metrics-certs\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:05.430773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.425177    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77dd41f-1fc2-4b1e-be96-902b2186b80e-tls-certs\") pod \"model-serving-api-75bd89fdfc-6zhqm\" (UID: \"e77dd41f-1fc2-4b1e-be96-902b2186b80e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:05.430773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.425237    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5388bcf-9195-4a03-89b5-d50dcb20b89d-cert\") pod \"kserve-controller-manager-776cc66d47-tbkh6\" (UID: \"d5388bcf-9195-4a03-89b5-d50dcb20b89d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:05.430773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.425279    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbd4\" (UniqueName: \"kubernetes.io/projected/37bc192f-69cc-4825-9a15-48bb8a6aae51-kube-api-access-vfbd4\") pod \"llmisvc-controller-manager-869d5dcc5d-jbr2p\" (UID: \"37bc192f-69cc-4825-9a15-48bb8a6aae51\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:05.443633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.443561    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 13:15:05.443995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.443808    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 13:15:05.443995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.443919    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-n97ks\""
Nov 03 13:15:05.527367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.526548    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n7th\" (UniqueName: \"kubernetes.io/projected/e77dd41f-1fc2-4b1e-be96-902b2186b80e-kube-api-access-9n7th\") pod \"model-serving-api-75bd89fdfc-6zhqm\" (UID: \"e77dd41f-1fc2-4b1e-be96-902b2186b80e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:05.527367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.527050    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2d78c7-0ebe-45dd-9683-02279bbfff20-metrics-certs\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:05.527367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.527114    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77dd41f-1fc2-4b1e-be96-902b2186b80e-tls-certs\") pod \"model-serving-api-75bd89fdfc-6zhqm\" (UID: \"e77dd41f-1fc2-4b1e-be96-902b2186b80e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:05.527367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.527139    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5388bcf-9195-4a03-89b5-d50dcb20b89d-cert\") pod \"kserve-controller-manager-776cc66d47-tbkh6\" (UID: \"d5388bcf-9195-4a03-89b5-d50dcb20b89d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:05.527367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.527180    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbd4\" (UniqueName: \"kubernetes.io/projected/37bc192f-69cc-4825-9a15-48bb8a6aae51-kube-api-access-vfbd4\") pod \"llmisvc-controller-manager-869d5dcc5d-jbr2p\" (UID: \"37bc192f-69cc-4825-9a15-48bb8a6aae51\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:05.527367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.527221    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37bc192f-69cc-4825-9a15-48bb8a6aae51-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-jbr2p\" (UID: \"37bc192f-69cc-4825-9a15-48bb8a6aae51\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:05.527367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.527258    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bggvb\" (UniqueName: \"kubernetes.io/projected/9a2d78c7-0ebe-45dd-9683-02279bbfff20-kube-api-access-bggvb\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:05.527367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.527309    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a2d78c7-0ebe-45dd-9683-02279bbfff20-cert\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:05.527367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.527367    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8v7f\" (UniqueName: \"kubernetes.io/projected/d5388bcf-9195-4a03-89b5-d50dcb20b89d-kube-api-access-j8v7f\") pod \"kserve-controller-manager-776cc66d47-tbkh6\" (UID: \"d5388bcf-9195-4a03-89b5-d50dcb20b89d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:05.529378 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:15:05.528535    4447 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Nov 03 13:15:05.529378 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:15:05.528624    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a2d78c7-0ebe-45dd-9683-02279bbfff20-cert podName:9a2d78c7-0ebe-45dd-9683-02279bbfff20 nodeName:}" failed. No retries permitted until 2026-11-03 13:15:06.028598967 +0000 UTC m=+393.840267133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9a2d78c7-0ebe-45dd-9683-02279bbfff20-cert") pod "odh-model-controller-769875db56-945dt" (UID: "9a2d78c7-0ebe-45dd-9683-02279bbfff20") : secret "odh-model-controller-webhook-cert" not found
Nov 03 13:15:05.531518 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.531465    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2d78c7-0ebe-45dd-9683-02279bbfff20-metrics-certs\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:05.531760 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.531720    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37bc192f-69cc-4825-9a15-48bb8a6aae51-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-jbr2p\" (UID: \"37bc192f-69cc-4825-9a15-48bb8a6aae51\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:05.532220 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.532190    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5388bcf-9195-4a03-89b5-d50dcb20b89d-cert\") pod \"kserve-controller-manager-776cc66d47-tbkh6\" (UID: \"d5388bcf-9195-4a03-89b5-d50dcb20b89d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:05.533076 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.533054    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77dd41f-1fc2-4b1e-be96-902b2186b80e-tls-certs\") pod \"model-serving-api-75bd89fdfc-6zhqm\" (UID: \"e77dd41f-1fc2-4b1e-be96-902b2186b80e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:05.559812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.559779    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8v7f\" (UniqueName: \"kubernetes.io/projected/d5388bcf-9195-4a03-89b5-d50dcb20b89d-kube-api-access-j8v7f\") pod \"kserve-controller-manager-776cc66d47-tbkh6\" (UID: \"d5388bcf-9195-4a03-89b5-d50dcb20b89d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:05.563879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.563825    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbd4\" (UniqueName: \"kubernetes.io/projected/37bc192f-69cc-4825-9a15-48bb8a6aae51-kube-api-access-vfbd4\") pod \"llmisvc-controller-manager-869d5dcc5d-jbr2p\" (UID: \"37bc192f-69cc-4825-9a15-48bb8a6aae51\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:05.563879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.563826    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bggvb\" (UniqueName: \"kubernetes.io/projected/9a2d78c7-0ebe-45dd-9683-02279bbfff20-kube-api-access-bggvb\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:05.564053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.563965    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n7th\" (UniqueName: \"kubernetes.io/projected/e77dd41f-1fc2-4b1e-be96-902b2186b80e-kube-api-access-9n7th\") pod \"model-serving-api-75bd89fdfc-6zhqm\" (UID: \"e77dd41f-1fc2-4b1e-be96-902b2186b80e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:05.691760 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.691719    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:05.739924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.739880    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:05.755776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.755738    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:05.978372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:05.978331    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-tbkh6"]
Nov 03 13:15:05.980687 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:15:05.980659    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5388bcf_9195_4a03_89b5_d50dcb20b89d.slice/crio-77ea25bce58df36a99b83c8df090da2b038ed7e2f5bebd3c25fabeb44682468e WatchSource:0}: Error finding container 77ea25bce58df36a99b83c8df090da2b038ed7e2f5bebd3c25fabeb44682468e: Status 404 returned error can't find the container with id 77ea25bce58df36a99b83c8df090da2b038ed7e2f5bebd3c25fabeb44682468e
Nov 03 13:15:06.031546 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:06.031508    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a2d78c7-0ebe-45dd-9683-02279bbfff20-cert\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:06.034444 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:06.034417    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a2d78c7-0ebe-45dd-9683-02279bbfff20-cert\") pod \"odh-model-controller-769875db56-945dt\" (UID: \"9a2d78c7-0ebe-45dd-9683-02279bbfff20\") " pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:06.047471 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:06.047444    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6zhqm"]
Nov 03 13:15:06.050000 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:15:06.049964    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77dd41f_1fc2_4b1e_be96_902b2186b80e.slice/crio-773d85b182eb31f6e81fc9f46fd9ab0329c02c626c9a179bbbfc8eb74658fabd WatchSource:0}: Error finding container 773d85b182eb31f6e81fc9f46fd9ab0329c02c626c9a179bbbfc8eb74658fabd: Status 404 returned error can't find the container with id 773d85b182eb31f6e81fc9f46fd9ab0329c02c626c9a179bbbfc8eb74658fabd
Nov 03 13:15:06.052446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:06.052425    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"]
Nov 03 13:15:06.054695 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:15:06.054672    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod37bc192f_69cc_4825_9a15_48bb8a6aae51.slice/crio-69dc2be3cc5ada80a8593419205556c155b0d79bdfe19e7a6e2634b2514f90bf WatchSource:0}: Error finding container 69dc2be3cc5ada80a8593419205556c155b0d79bdfe19e7a6e2634b2514f90bf: Status 404 returned error can't find the container with id 69dc2be3cc5ada80a8593419205556c155b0d79bdfe19e7a6e2634b2514f90bf
Nov 03 13:15:06.080016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:06.079934    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:06.318943 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:06.318905    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6" event={"ID":"d5388bcf-9195-4a03-89b5-d50dcb20b89d","Type":"ContainerStarted","Data":"77ea25bce58df36a99b83c8df090da2b038ed7e2f5bebd3c25fabeb44682468e"}
Nov 03 13:15:06.319805 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:06.319777    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p" event={"ID":"37bc192f-69cc-4825-9a15-48bb8a6aae51","Type":"ContainerStarted","Data":"69dc2be3cc5ada80a8593419205556c155b0d79bdfe19e7a6e2634b2514f90bf"}
Nov 03 13:15:06.321075 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:06.321039    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm" event={"ID":"e77dd41f-1fc2-4b1e-be96-902b2186b80e","Type":"ContainerStarted","Data":"773d85b182eb31f6e81fc9f46fd9ab0329c02c626c9a179bbbfc8eb74658fabd"}
Nov 03 13:15:06.366395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:06.366369    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-945dt"]
Nov 03 13:15:06.368869 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:15:06.368832    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2d78c7_0ebe_45dd_9683_02279bbfff20.slice/crio-21c0f3c7c2b0e32021ef0e9f84fe399b19df533c684b82fb6aba6451bab8839f WatchSource:0}: Error finding container 21c0f3c7c2b0e32021ef0e9f84fe399b19df533c684b82fb6aba6451bab8839f: Status 404 returned error can't find the container with id 21c0f3c7c2b0e32021ef0e9f84fe399b19df533c684b82fb6aba6451bab8839f
Nov 03 13:15:07.254390 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.253199    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:15:07.254390 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.254230    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:15:07.310758 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.310669    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5776b58b5b-q6zgl"]
Nov 03 13:15:07.337487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.337438    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5776b58b5b-q6zgl" podUID="a67aa6d1-046f-4aa1-8a48-5379c8d8ba05" containerName="manager" containerID="cri-o://a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272" gracePeriod=30
Nov 03 13:15:07.337913 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.337756    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-945dt" event={"ID":"9a2d78c7-0ebe-45dd-9683-02279bbfff20","Type":"ContainerStarted","Data":"21c0f3c7c2b0e32021ef0e9f84fe399b19df533c684b82fb6aba6451bab8839f"}
Nov 03 13:15:07.791137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.791105    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:15:07.863559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.863484    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-kube-api-access-jnsck\" (UniqueName: \"kubernetes.io/projected/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-kube-api-access-jnsck\") pod \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") "
Nov 03 13:15:07.863559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.863555    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-metrics-certs\") pod \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") "
Nov 03 13:15:07.863774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.863638    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-webhook-cert\") pod \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\" (UID: \"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05\") "
Nov 03 13:15:07.876718 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.876674    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-webhook-cert" pod "a67aa6d1-046f-4aa1-8a48-5379c8d8ba05" (UID: "a67aa6d1-046f-4aa1-8a48-5379c8d8ba05"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:07.877985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.877946    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-kube-api-access-jnsck" pod "a67aa6d1-046f-4aa1-8a48-5379c8d8ba05" (UID: "a67aa6d1-046f-4aa1-8a48-5379c8d8ba05"). InnerVolumeSpecName "kube-api-access-jnsck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:07.893243 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.887808    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-metrics-certs" pod "a67aa6d1-046f-4aa1-8a48-5379c8d8ba05" (UID: "a67aa6d1-046f-4aa1-8a48-5379c8d8ba05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:07.964579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.964541    4447 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-webhook-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:07.964579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.964582    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsck\" (UniqueName: \"kubernetes.io/projected/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-kube-api-access-jnsck\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:07.964831 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:07.964597    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.359471 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:08.359418    4447 generic.go:356] "Generic (PLEG): container finished" podID="a67aa6d1-046f-4aa1-8a48-5379c8d8ba05" containerID="a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272" exitCode=0
Nov 03 13:15:08.359935 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:08.359570    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5776b58b5b-q6zgl" event={"ID":"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05","Type":"ContainerDied","Data":"a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272"}
Nov 03 13:15:08.359935 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:08.359631    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5776b58b5b-q6zgl" event={"ID":"a67aa6d1-046f-4aa1-8a48-5379c8d8ba05","Type":"ContainerDied","Data":"6ef7a330bf86a49dc071df530681e4a0cb8e823ec5ad181db3cb166653baa83c"}
Nov 03 13:15:08.359935 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:08.359667    4447 scope.go:122] "RemoveContainer" containerID="a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272"
Nov 03 13:15:08.359935 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:08.359832    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5776b58b5b-q6zgl"
Nov 03 13:15:08.375197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:08.375166    4447 scope.go:122] "RemoveContainer" containerID="a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272"
Nov 03 13:15:08.375689 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:15:08.375664    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272\": container with ID starting with a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272 not found: ID does not exist" containerID="a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272"
Nov 03 13:15:08.375797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:08.375701    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272"} err="failed to get container status \"a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272\": rpc error: code = NotFound desc = could not find container \"a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272\": container with ID starting with a2e0b571519e074425a3cd2ad5ad5ec512527995682baf30e9b7319ccba63272 not found: ID does not exist"
Nov 03 13:15:08.381160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:08.381132    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5776b58b5b-q6zgl"]
Nov 03 13:15:08.384990 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:08.384968    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5776b58b5b-q6zgl"]
Nov 03 13:15:10.321273 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.320793    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67aa6d1-046f-4aa1-8a48-5379c8d8ba05" path="/var/lib/kubelet/pods/a67aa6d1-046f-4aa1-8a48-5379c8d8ba05/volumes"
Nov 03 13:15:10.377684 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.377641    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6" event={"ID":"d5388bcf-9195-4a03-89b5-d50dcb20b89d","Type":"ContainerStarted","Data":"3e1655a926ebc6843f58060dbfe02e617cdc4a7982da3e22b328d54985158dfb"}
Nov 03 13:15:10.378210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.378193    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:10.379827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.379800    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p" event={"ID":"37bc192f-69cc-4825-9a15-48bb8a6aae51","Type":"ContainerStarted","Data":"07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822"}
Nov 03 13:15:10.380390 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.380338    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:10.383210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.382621    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm" event={"ID":"e77dd41f-1fc2-4b1e-be96-902b2186b80e","Type":"ContainerStarted","Data":"2a3710b610af603143ab94a42cbec34b48f712e102b49c7ef168af6c7ca4c309"}
Nov 03 13:15:10.383210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.383132    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:10.399134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.397738    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6" podStartSLOduration=1.375129425 podStartE2EDuration="5.397720423s" totalImagesPullingTime="4.022590998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:10.395208004 +0000 UTC m=+398.206876210" watchObservedRunningTime="2026-11-03 13:15:10.397720423 +0000 UTC m=+398.209388625"
Nov 03 13:15:10.411731 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.411219    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm" podStartSLOduration=1.470105054 podStartE2EDuration="5.411205722s" totalImagesPullingTime="3.941100668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:10.410874042 +0000 UTC m=+398.222542229" watchObservedRunningTime="2026-11-03 13:15:10.411205722 +0000 UTC m=+398.222873911"
Nov 03 13:15:10.428565 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:10.426864    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p" podStartSLOduration=1.486572426 podStartE2EDuration="5.42684551s" totalImagesPullingTime="3.940273084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:10.42474574 +0000 UTC m=+398.236413962" watchObservedRunningTime="2026-11-03 13:15:10.42684551 +0000 UTC m=+398.238513700"
Nov 03 13:15:11.390803 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:11.390721    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-945dt" event={"ID":"9a2d78c7-0ebe-45dd-9683-02279bbfff20","Type":"ContainerStarted","Data":"a396f49df35ec50b7bae280bf348ba6fda30362fbf4983ce92bfe6ff933a2b22"}
Nov 03 13:15:11.391260 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:11.390939    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:11.408144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:11.408097    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-945dt" podStartSLOduration=2.016417663 podStartE2EDuration="6.408084094s" totalImagesPullingTime="4.391666431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:11.40624739 +0000 UTC m=+399.217915597" watchObservedRunningTime="2026-11-03 13:15:11.408084094 +0000 UTC m=+399.219752317"
Nov 03 13:15:12.397985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:12.397942    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw" event={"ID":"63ddba5e-73fe-472c-bff2-e98dafa5977a","Type":"ContainerStarted","Data":"32f3a75028eedb519443df828b6b30d0e862f85ca4e300cf695161dfb6e7de4c"}
Nov 03 13:15:12.414111 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:12.414056    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw" podStartSLOduration=2.308100558 podStartE2EDuration="16.414039108s" totalImagesPullingTime="14.10593855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:12.412368854 +0000 UTC m=+400.224037040" watchObservedRunningTime="2026-11-03 13:15:12.414039108 +0000 UTC m=+400.225707296"
Nov 03 13:15:13.403104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:13.403073    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:15:22.400477 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:22.400402    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-945dt"
Nov 03 13:15:22.402288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:22.402264    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-6zhqm"
Nov 03 13:15:24.414142 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:24.414113    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2trjw"
Nov 03 13:15:33.410557 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:15:33.410513    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:15:38.243252 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.243210    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rj2vw"]
Nov 03 13:15:38.243802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.243630    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a67aa6d1-046f-4aa1-8a48-5379c8d8ba05" containerName="manager"
Nov 03 13:15:38.243802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.243642    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67aa6d1-046f-4aa1-8a48-5379c8d8ba05" containerName="manager"
Nov 03 13:15:38.243802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.243710    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67aa6d1-046f-4aa1-8a48-5379c8d8ba05" containerName="manager"
Nov 03 13:15:38.372551 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.372520    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rj2vw"
Nov 03 13:15:38.376045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.376006    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 13:15:38.376200 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.376064    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 13:15:38.379606 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.379581    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rj2vw"]
Nov 03 13:15:38.379718 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.379636    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rj2vw"]
Nov 03 13:15:38.415511 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:15:38.415469    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6kgbj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-rj2vw" podUID="29fd9ec0-ed46-4aa9-83c3-1a5913831681"
Nov 03 13:15:38.440003 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.439969    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kgbj\" (UniqueName: \"kubernetes.io/projected/29fd9ec0-ed46-4aa9-83c3-1a5913831681-kube-api-access-6kgbj\") pod \"authorino-6f6944bcdd-rj2vw\" (UID: \"29fd9ec0-ed46-4aa9-83c3-1a5913831681\") " pod="kuadrant-system/authorino-6f6944bcdd-rj2vw"
Nov 03 13:15:38.444423 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.444393    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kk72q\""
Nov 03 13:15:38.467434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.467404    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"]
Nov 03 13:15:38.473691 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.473670    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.483194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.483008    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"]
Nov 03 13:15:38.539494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.539426    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rj2vw"
Nov 03 13:15:38.541248 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.541226    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgbj\" (UniqueName: \"kubernetes.io/projected/29fd9ec0-ed46-4aa9-83c3-1a5913831681-kube-api-access-6kgbj\") pod \"authorino-6f6944bcdd-rj2vw\" (UID: \"29fd9ec0-ed46-4aa9-83c3-1a5913831681\") " pod="kuadrant-system/authorino-6f6944bcdd-rj2vw"
Nov 03 13:15:38.541377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.541279    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-metrics-certs\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.541377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.541328    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-maas-api-tls\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.541483 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.541417    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e448c30a-a1f7-4d25-9a95-0005c1587096-openshift-service-ca\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.541483 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.541472    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5vg\" (UniqueName: \"kubernetes.io/projected/e448c30a-a1f7-4d25-9a95-0005c1587096-kube-api-access-2d5vg\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.544663 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.544640    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p5qd5\""
Nov 03 13:15:38.544773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.544759    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 13:15:38.544830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.544820    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 13:15:38.548830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.548808    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kgbj\" (UniqueName: \"kubernetes.io/projected/29fd9ec0-ed46-4aa9-83c3-1a5913831681-kube-api-access-6kgbj\") pod \"authorino-6f6944bcdd-rj2vw\" (UID: \"29fd9ec0-ed46-4aa9-83c3-1a5913831681\") " pod="kuadrant-system/authorino-6f6944bcdd-rj2vw"
Nov 03 13:15:38.550003 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.549989    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rj2vw"
Nov 03 13:15:38.642213 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.642179    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fd9ec0-ed46-4aa9-83c3-1a5913831681-kube-api-access-6kgbj\" (UniqueName: \"kubernetes.io/projected/29fd9ec0-ed46-4aa9-83c3-1a5913831681-kube-api-access-6kgbj\") pod \"29fd9ec0-ed46-4aa9-83c3-1a5913831681\" (UID: \"29fd9ec0-ed46-4aa9-83c3-1a5913831681\") "
Nov 03 13:15:38.642436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.642311    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-metrics-certs\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.642436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.642360    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-maas-api-tls\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.642436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.642381    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e448c30a-a1f7-4d25-9a95-0005c1587096-openshift-service-ca\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.642436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.642402    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5vg\" (UniqueName: \"kubernetes.io/projected/e448c30a-a1f7-4d25-9a95-0005c1587096-kube-api-access-2d5vg\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.643088 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.643065    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e448c30a-a1f7-4d25-9a95-0005c1587096-openshift-service-ca\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.644463 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.644434    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fd9ec0-ed46-4aa9-83c3-1a5913831681-kube-api-access-6kgbj" pod "29fd9ec0-ed46-4aa9-83c3-1a5913831681" (UID: "29fd9ec0-ed46-4aa9-83c3-1a5913831681"). InnerVolumeSpecName "kube-api-access-6kgbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:38.644826 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.644806    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-metrics-certs\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.644892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.644867    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-maas-api-tls\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.654420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.654397    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5vg\" (UniqueName: \"kubernetes.io/projected/e448c30a-a1f7-4d25-9a95-0005c1587096-kube-api-access-2d5vg\") pod \"maas-api-97b845c85-s6sxx\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:38.743830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.743793    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kgbj\" (UniqueName: \"kubernetes.io/projected/29fd9ec0-ed46-4aa9-83c3-1a5913831681-kube-api-access-6kgbj\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:38.794029 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:38.793946    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:39.105545 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:39.105444    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"]
Nov 03 13:15:39.108476 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:15:39.108450    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode448c30a_a1f7_4d25_9a95_0005c1587096.slice/crio-bbf21a65560cbcd0566b777ea809ab5b1b3848af8ab61359d1de0068d5805cff WatchSource:0}: Error finding container bbf21a65560cbcd0566b777ea809ab5b1b3848af8ab61359d1de0068d5805cff: Status 404 returned error can't find the container with id bbf21a65560cbcd0566b777ea809ab5b1b3848af8ab61359d1de0068d5805cff
Nov 03 13:15:39.545395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:39.545368    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rj2vw"
Nov 03 13:15:39.545395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:39.545377    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx" event={"ID":"e448c30a-a1f7-4d25-9a95-0005c1587096","Type":"ContainerStarted","Data":"bbf21a65560cbcd0566b777ea809ab5b1b3848af8ab61359d1de0068d5805cff"}
Nov 03 13:15:39.586664 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:39.586602    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rj2vw"]
Nov 03 13:15:39.590933 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:39.590905    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rj2vw"]
Nov 03 13:15:40.320294 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:40.320252    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fd9ec0-ed46-4aa9-83c3-1a5913831681" path="/var/lib/kubelet/pods/29fd9ec0-ed46-4aa9-83c3-1a5913831681/volumes"
Nov 03 13:15:41.397381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:41.397298    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-tbkh6"
Nov 03 13:15:41.558433 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:41.558388    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx" event={"ID":"e448c30a-a1f7-4d25-9a95-0005c1587096","Type":"ContainerStarted","Data":"46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac"}
Nov 03 13:15:41.558620 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:41.558464    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:15:41.577522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:41.577469    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx" podStartSLOduration=1.656267863 podStartE2EDuration="3.577453749s" totalImagesPullingTime="1.921185886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:41.576942169 +0000 UTC m=+429.388610357" watchObservedRunningTime="2026-11-03 13:15:41.577453749 +0000 UTC m=+429.389121931"
Nov 03 13:15:42.400414 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:42.400386    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:15:47.568861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:15:47.568829    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 13:16:02.096281 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.096243    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"]
Nov 03 13:16:02.105238 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.105218    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:02.108879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.108783    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"]
Nov 03 13:16:02.155549 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.155511    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b0996d1-8bb0-499c-b88e-b334e4deb0f6-cert\") pod \"llmisvc-controller-manager-6466c554d8-p8b4c\" (UID: \"5b0996d1-8bb0-499c-b88e-b334e4deb0f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:02.155706 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.155561    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppt97\" (UniqueName: \"kubernetes.io/projected/5b0996d1-8bb0-499c-b88e-b334e4deb0f6-kube-api-access-ppt97\") pod \"llmisvc-controller-manager-6466c554d8-p8b4c\" (UID: \"5b0996d1-8bb0-499c-b88e-b334e4deb0f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:02.256048 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.256004    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b0996d1-8bb0-499c-b88e-b334e4deb0f6-cert\") pod \"llmisvc-controller-manager-6466c554d8-p8b4c\" (UID: \"5b0996d1-8bb0-499c-b88e-b334e4deb0f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:02.256226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.256056    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppt97\" (UniqueName: \"kubernetes.io/projected/5b0996d1-8bb0-499c-b88e-b334e4deb0f6-kube-api-access-ppt97\") pod \"llmisvc-controller-manager-6466c554d8-p8b4c\" (UID: \"5b0996d1-8bb0-499c-b88e-b334e4deb0f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:02.258478 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.258453    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b0996d1-8bb0-499c-b88e-b334e4deb0f6-cert\") pod \"llmisvc-controller-manager-6466c554d8-p8b4c\" (UID: \"5b0996d1-8bb0-499c-b88e-b334e4deb0f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:02.263372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.263330    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppt97\" (UniqueName: \"kubernetes.io/projected/5b0996d1-8bb0-499c-b88e-b334e4deb0f6-kube-api-access-ppt97\") pod \"llmisvc-controller-manager-6466c554d8-p8b4c\" (UID: \"5b0996d1-8bb0-499c-b88e-b334e4deb0f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:02.419382 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.419279    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:02.693269 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:02.693244    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"]
Nov 03 13:16:02.695594 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:16:02.695565    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0996d1_8bb0_499c_b88e_b334e4deb0f6.slice/crio-7ef71706db3f6e29d480727be4ca06a3158b29212fe2d80a4320c0f6170178ce WatchSource:0}: Error finding container 7ef71706db3f6e29d480727be4ca06a3158b29212fe2d80a4320c0f6170178ce: Status 404 returned error can't find the container with id 7ef71706db3f6e29d480727be4ca06a3158b29212fe2d80a4320c0f6170178ce
Nov 03 13:16:03.410864 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:16:03.410780    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:16:03.684511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:03.684474    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c" event={"ID":"5b0996d1-8bb0-499c-b88e-b334e4deb0f6","Type":"ContainerStarted","Data":"b4edc587d27c3c32b2225b10f0db0a0efd8c98adec5ffddcdf506c0e9c063f31"}
Nov 03 13:16:03.684694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:03.684527    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c" event={"ID":"5b0996d1-8bb0-499c-b88e-b334e4deb0f6","Type":"ContainerStarted","Data":"7ef71706db3f6e29d480727be4ca06a3158b29212fe2d80a4320c0f6170178ce"}
Nov 03 13:16:03.684694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:03.684587    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:03.701334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:03.701281    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c" podStartSLOduration=1.32111322 podStartE2EDuration="1.701267813s" totalImagesPullingTime="380.154593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:03.698693157 +0000 UTC m=+451.510361357" watchObservedRunningTime="2026-11-03 13:16:03.701267813 +0000 UTC m=+451.512935979"
Nov 03 13:16:33.410576 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:16:33.410536    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:16:34.691516 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:34.691481    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p8b4c"
Nov 03 13:16:34.731659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:34.731626    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"]
Nov 03 13:16:34.731901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:34.731860    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p" podUID="37bc192f-69cc-4825-9a15-48bb8a6aae51" containerName="manager" containerID="cri-o://07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822" gracePeriod=30
Nov 03 13:16:34.977889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:34.977862    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:16:35.012491 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.012464    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bc192f-69cc-4825-9a15-48bb8a6aae51-kube-api-access-vfbd4\" (UniqueName: \"kubernetes.io/projected/37bc192f-69cc-4825-9a15-48bb8a6aae51-kube-api-access-vfbd4\") pod \"37bc192f-69cc-4825-9a15-48bb8a6aae51\" (UID: \"37bc192f-69cc-4825-9a15-48bb8a6aae51\") "
Nov 03 13:16:35.012674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.012605    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37bc192f-69cc-4825-9a15-48bb8a6aae51-cert\" (UniqueName: \"kubernetes.io/secret/37bc192f-69cc-4825-9a15-48bb8a6aae51-cert\") pod \"37bc192f-69cc-4825-9a15-48bb8a6aae51\" (UID: \"37bc192f-69cc-4825-9a15-48bb8a6aae51\") "
Nov 03 13:16:35.014577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.014544    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bc192f-69cc-4825-9a15-48bb8a6aae51-kube-api-access-vfbd4" pod "37bc192f-69cc-4825-9a15-48bb8a6aae51" (UID: "37bc192f-69cc-4825-9a15-48bb8a6aae51"). InnerVolumeSpecName "kube-api-access-vfbd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:35.014729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.014710    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37bc192f-69cc-4825-9a15-48bb8a6aae51-cert" pod "37bc192f-69cc-4825-9a15-48bb8a6aae51" (UID: "37bc192f-69cc-4825-9a15-48bb8a6aae51"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:35.113713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.113681    4447 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37bc192f-69cc-4825-9a15-48bb8a6aae51-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:35.113713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.113707    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfbd4\" (UniqueName: \"kubernetes.io/projected/37bc192f-69cc-4825-9a15-48bb8a6aae51-kube-api-access-vfbd4\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:35.848182 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.848143    4447 generic.go:356] "Generic (PLEG): container finished" podID="37bc192f-69cc-4825-9a15-48bb8a6aae51" containerID="07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822" exitCode=0
Nov 03 13:16:35.848656 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.848194    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"
Nov 03 13:16:35.848656 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.848195    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p" event={"ID":"37bc192f-69cc-4825-9a15-48bb8a6aae51","Type":"ContainerDied","Data":"07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822"}
Nov 03 13:16:35.848656 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.848301    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p" event={"ID":"37bc192f-69cc-4825-9a15-48bb8a6aae51","Type":"ContainerDied","Data":"69dc2be3cc5ada80a8593419205556c155b0d79bdfe19e7a6e2634b2514f90bf"}
Nov 03 13:16:35.848656 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.848329    4447 scope.go:122] "RemoveContainer" containerID="07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822"
Nov 03 13:16:35.857588 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.857565    4447 scope.go:122] "RemoveContainer" containerID="07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822"
Nov 03 13:16:35.857868 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:16:35.857842    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822\": container with ID starting with 07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822 not found: ID does not exist" containerID="07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822"
Nov 03 13:16:35.857948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.857875    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822"} err="failed to get container status \"07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822\": rpc error: code = NotFound desc = could not find container \"07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822\": container with ID starting with 07c57cdaef549bff783341e4023e7365b99365bd2771d630710840982bdd6822 not found: ID does not exist"
Nov 03 13:16:35.866982 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.866958    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"]
Nov 03 13:16:35.869967 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:35.869947    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jbr2p"]
Nov 03 13:16:36.317809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:16:36.317768    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bc192f-69cc-4825-9a15-48bb8a6aae51" path="/var/lib/kubelet/pods/37bc192f-69cc-4825-9a15-48bb8a6aae51/volumes"
Nov 03 13:17:03.410732 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:17:03.410696    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:17:23.094190 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:23.094158    4447 generic.go:356] "Generic (PLEG): container finished" podID="72d09d90-d90c-4afe-bd2f-c95980f250e8" containerID="0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67" exitCode=1
Nov 03 13:17:23.094641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:23.094224    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-6g9s8" event={"ID":"72d09d90-d90c-4afe-bd2f-c95980f250e8","Type":"ContainerDied","Data":"0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67"}
Nov 03 13:17:23.094641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:23.094565    4447 scope.go:122] "RemoveContainer" containerID="0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67"
Nov 03 13:17:24.018261 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.018230    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"]
Nov 03 13:17:24.018766 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.018742    4447 cpu_manager.go:414] "containerMap: removing container" podUID="37bc192f-69cc-4825-9a15-48bb8a6aae51" containerName="manager"
Nov 03 13:17:24.018766 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.018761    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bc192f-69cc-4825-9a15-48bb8a6aae51" containerName="manager"
Nov 03 13:17:24.018955 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.018872    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bc192f-69cc-4825-9a15-48bb8a6aae51" containerName="manager"
Nov 03 13:17:24.022354 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.022318    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.029886 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.028703    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 13:17:24.042927 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.042900    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"]
Nov 03 13:17:24.101397 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.101339    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-6g9s8" event={"ID":"72d09d90-d90c-4afe-bd2f-c95980f250e8","Type":"ContainerStarted","Data":"b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf"}
Nov 03 13:17:24.101797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.101580    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:17:24.117616 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.117582    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nxhjw\""
Nov 03 13:17:24.145226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.145194    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.145437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.145243    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53934dd9-9712-44e9-883b-2693539dc602-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.145437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.145278    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53934dd9-9712-44e9-883b-2693539dc602-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.145437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.145363    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.145604 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.145446    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.145604 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.145477    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.145604 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.145541    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54pr\" (UniqueName: \"kubernetes.io/projected/53934dd9-9712-44e9-883b-2693539dc602-kube-api-access-g54pr\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.145711 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.145654    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.145711 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.145689    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53934dd9-9712-44e9-883b-2693539dc602-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.246904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.246859    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54pr\" (UniqueName: \"kubernetes.io/projected/53934dd9-9712-44e9-883b-2693539dc602-kube-api-access-g54pr\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.246925    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.246951    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53934dd9-9712-44e9-883b-2693539dc602-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247013    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247046    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53934dd9-9712-44e9-883b-2693539dc602-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247071    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53934dd9-9712-44e9-883b-2693539dc602-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247096    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247133    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247159    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247663 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247630    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247738 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247664    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.247932 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247904    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.248013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.247966    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53934dd9-9712-44e9-883b-2693539dc602-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.248013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.248001    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.249858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.249827    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53934dd9-9712-44e9-883b-2693539dc602-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.250082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.250063    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53934dd9-9712-44e9-883b-2693539dc602-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.257635 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.257608    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53934dd9-9712-44e9-883b-2693539dc602-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.257873 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.257853    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54pr\" (UniqueName: \"kubernetes.io/projected/53934dd9-9712-44e9-883b-2693539dc602-kube-api-access-g54pr\") pod \"maas-default-gateway-openshift-default-84587756dc-sl776\" (UID: \"53934dd9-9712-44e9-883b-2693539dc602\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.336362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.336269    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:24.626097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:24.626070    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"]
Nov 03 13:17:24.629109 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:17:24.629082    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53934dd9_9712_44e9_883b_2693539dc602.slice/crio-f17b0d8f9de3c463900b9d25d1e68566c6a359de7790e0386cdab267d6039eac WatchSource:0}: Error finding container f17b0d8f9de3c463900b9d25d1e68566c6a359de7790e0386cdab267d6039eac: Status 404 returned error can't find the container with id f17b0d8f9de3c463900b9d25d1e68566c6a359de7790e0386cdab267d6039eac
Nov 03 13:17:25.108268 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:25.108234    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" event={"ID":"53934dd9-9712-44e9-883b-2693539dc602","Type":"ContainerStarted","Data":"f17b0d8f9de3c463900b9d25d1e68566c6a359de7790e0386cdab267d6039eac"}
Nov 03 13:17:27.826488 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:27.826454    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:17:27.826892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:27.826530    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:17:27.826892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:27.826563    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:17:28.130099 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:28.130017    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" event={"ID":"53934dd9-9712-44e9-883b-2693539dc602","Type":"ContainerStarted","Data":"48961a53bf6f0e0417e8f62e27b0416e78f14ddf66269778665c7708906e0203"}
Nov 03 13:17:28.150274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:28.150221    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podStartSLOduration=1.955832795 podStartE2EDuration="5.150206556s" totalImagesPullingTime="3.194373761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:28.148534375 +0000 UTC m=+535.960202623" watchObservedRunningTime="2026-11-03 13:17:28.150206556 +0000 UTC m=+535.961874743"
Nov 03 13:17:28.338365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:28.338109    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:28.338365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:28.338232    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:17:28.338365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:28.338289    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 13:17:29.339364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:29.338754    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:17:29.339364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:29.338820    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 13:17:30.339370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:30.338736    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:17:30.339370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:30.338795    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 13:17:31.338776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:31.338736    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:17:31.339081 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:31.339056    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 13:17:32.339720 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:32.339681    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:17:32.340264 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:32.340234    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 13:17:33.342369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:33.340842    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:17:33.342369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:33.340924    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 13:17:33.411001 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:17:33.410963    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:17:34.337496 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:34.337457    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:17:34.337672 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:34.337533    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 13:17:35.111398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:35.111370    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:17:36.082578 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:36.082537    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:36.082963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:36.082936    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:36.083833 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:36.083811    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776"
Nov 03 13:17:39.039635 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.039599    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"]
Nov 03 13:17:39.045474 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.045450    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.058039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.058011    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"]
Nov 03 13:17:39.186137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.186099    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.186316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.186162    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c35106c-e68e-4d61-ad65-88b96270b873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.186316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.186226    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzfr\" (UniqueName: \"kubernetes.io/projected/6c35106c-e68e-4d61-ad65-88b96270b873-kube-api-access-xdzfr\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.186316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.186283    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.186316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.186307    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c35106c-e68e-4d61-ad65-88b96270b873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.186557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.186408    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.186557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.186467    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.186557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.186515    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c35106c-e68e-4d61-ad65-88b96270b873-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.186753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.186622    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.287649 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.287614    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c35106c-e68e-4d61-ad65-88b96270b873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.287649 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.287651    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzfr\" (UniqueName: \"kubernetes.io/projected/6c35106c-e68e-4d61-ad65-88b96270b873-kube-api-access-xdzfr\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.287902 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.287678    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.287902 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.287693    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c35106c-e68e-4d61-ad65-88b96270b873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.287902 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.287809    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.287902 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.287847    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.287902 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.287879    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c35106c-e68e-4d61-ad65-88b96270b873-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.288169 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.287908    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.288169 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.287964    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.288169 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.288084    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.288336 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.288208    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.288431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.288370    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.288543 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.288451    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.288543 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.288510    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c35106c-e68e-4d61-ad65-88b96270b873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.290138 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.290086    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c35106c-e68e-4d61-ad65-88b96270b873-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.290392 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.290372    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c35106c-e68e-4d61-ad65-88b96270b873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.294694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.294671    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c35106c-e68e-4d61-ad65-88b96270b873-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.294860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.294840    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzfr\" (UniqueName: \"kubernetes.io/projected/6c35106c-e68e-4d61-ad65-88b96270b873-kube-api-access-xdzfr\") pod \"maas-default-gateway-openshift-default-84587756dc-mnk7k\" (UID: \"6c35106c-e68e-4d61-ad65-88b96270b873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.364339 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.364293    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:39.648138 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.648115    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"]
Nov 03 13:17:39.650329 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:17:39.650298    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c35106c_e68e_4d61_ad65_88b96270b873.slice/crio-635f4217393c47bf5488cec8b054585e4558f2bd7a13f55f7057ee374e0ce747 WatchSource:0}: Error finding container 635f4217393c47bf5488cec8b054585e4558f2bd7a13f55f7057ee374e0ce747: Status 404 returned error can't find the container with id 635f4217393c47bf5488cec8b054585e4558f2bd7a13f55f7057ee374e0ce747
Nov 03 13:17:39.652977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.652943    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:17:39.653072 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.653007    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:17:39.653072 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:39.653041    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:17:40.213396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:40.213335    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" event={"ID":"6c35106c-e68e-4d61-ad65-88b96270b873","Type":"ContainerStarted","Data":"f5950e034840531c7a3286a4d605c44d906ace848b09a843c1bdf079c7fe0407"}
Nov 03 13:17:40.213796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:40.213429    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" event={"ID":"6c35106c-e68e-4d61-ad65-88b96270b873","Type":"ContainerStarted","Data":"635f4217393c47bf5488cec8b054585e4558f2bd7a13f55f7057ee374e0ce747"}
Nov 03 13:17:40.232051 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:40.232004    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podStartSLOduration=1.231992024 podStartE2EDuration="1.231992024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:40.230803526 +0000 UTC m=+548.042471732" watchObservedRunningTime="2026-11-03 13:17:40.231992024 +0000 UTC m=+548.043660211"
Nov 03 13:17:40.364704 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:40.364670    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:40.366211 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:40.366186    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:17:40.366291 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:40.366240    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 13:17:41.364831 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:41.364797    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:17:41.365232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:41.364849    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 13:17:42.364887 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:42.364850    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:17:42.365429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:42.364935    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 13:17:43.364889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:43.364846    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:17:43.365107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:43.364906    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 13:17:44.365406 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:44.365300    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:17:44.365406 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:44.365382    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 13:17:45.365460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:45.365420    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:17:45.365857 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:45.365476    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 13:17:46.365758 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:46.365701    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:17:46.366283 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:46.365782    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 13:17:48.364938 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:48.364894    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:17:48.365451 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:48.364961    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:17:49.406503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:49.406459    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:17:49.406980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:49.406516    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:17:49.966064 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:49.966035    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:49.966329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:49.966306    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:49.967052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:49.967032    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k"
Nov 03 13:17:54.072398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.072362    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"]
Nov 03 13:17:54.076188 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.076162    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.091207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.091178    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"]
Nov 03 13:17:54.248305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.248263    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.248305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.248302    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08586908-beb4-471e-8071-0ffc382d6ac2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.248535 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.248324    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08586908-beb4-471e-8071-0ffc382d6ac2-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.248535 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.248427    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.248535 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.248460    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.248651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.248550    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.248651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.248571    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.248651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.248617    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08586908-beb4-471e-8071-0ffc382d6ac2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.248751 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.248701    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5xvh\" (UniqueName: \"kubernetes.io/projected/08586908-beb4-471e-8071-0ffc382d6ac2-kube-api-access-z5xvh\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349123    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08586908-beb4-471e-8071-0ffc382d6ac2-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349159    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349177    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349210    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349227    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349274    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08586908-beb4-471e-8071-0ffc382d6ac2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349299    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5xvh\" (UniqueName: \"kubernetes.io/projected/08586908-beb4-471e-8071-0ffc382d6ac2-kube-api-access-z5xvh\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349448    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349475    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08586908-beb4-471e-8071-0ffc382d6ac2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349819 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349617    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349819 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349709    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349819 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349728    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.349931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.349893    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.350168 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.350149    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08586908-beb4-471e-8071-0ffc382d6ac2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.351700 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.351677    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08586908-beb4-471e-8071-0ffc382d6ac2-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.351794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.351743    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08586908-beb4-471e-8071-0ffc382d6ac2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.356846 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.356819    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08586908-beb4-471e-8071-0ffc382d6ac2-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.356949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.356933    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5xvh\" (UniqueName: \"kubernetes.io/projected/08586908-beb4-471e-8071-0ffc382d6ac2-kube-api-access-z5xvh\") pod \"maas-default-gateway-openshift-default-84587756dc-z544g\" (UID: \"08586908-beb4-471e-8071-0ffc382d6ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.391070 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.391043    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:54.668582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.668434    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"]
Nov 03 13:17:54.671552 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:17:54.671526    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08586908_beb4_471e_8071_0ffc382d6ac2.slice/crio-fe4693ad55972f245d4e163fef701b7424858213986f45650ebde4e73ed8451c WatchSource:0}: Error finding container fe4693ad55972f245d4e163fef701b7424858213986f45650ebde4e73ed8451c: Status 404 returned error can't find the container with id fe4693ad55972f245d4e163fef701b7424858213986f45650ebde4e73ed8451c
Nov 03 13:17:54.674113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.674081    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:17:54.674178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.674146    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:17:54.674215 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:54.674178    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:17:55.315042 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:55.315006    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" event={"ID":"08586908-beb4-471e-8071-0ffc382d6ac2","Type":"ContainerStarted","Data":"7b5dcb921d6cfefe33d19c3681110aadab0341c0d7bc7f757d57e7f1bfad9b87"}
Nov 03 13:17:55.315513 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:55.315055    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" event={"ID":"08586908-beb4-471e-8071-0ffc382d6ac2","Type":"ContainerStarted","Data":"fe4693ad55972f245d4e163fef701b7424858213986f45650ebde4e73ed8451c"}
Nov 03 13:17:55.332666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:55.332608    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" podStartSLOduration=1.3325962040000001 podStartE2EDuration="1.332596204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:55.331864614 +0000 UTC m=+563.143532820" watchObservedRunningTime="2026-11-03 13:17:55.332596204 +0000 UTC m=+563.144264454"
Nov 03 13:17:55.391784 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:55.391752    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:17:55.393206 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:55.393185    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-z544g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 13:17:55.393275 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:55.393233    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" podUID="08586908-beb4-471e-8071-0ffc382d6ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 13:17:56.391875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:56.391842    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-z544g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 13:17:56.392393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:56.391926    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" podUID="08586908-beb4-471e-8071-0ffc382d6ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 13:17:57.391472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:57.391435    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-z544g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 13:17:57.391665 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:57.391487    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" podUID="08586908-beb4-471e-8071-0ffc382d6ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 13:17:58.392070 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:58.392035    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-z544g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 13:17:58.392490 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:58.392094    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" podUID="08586908-beb4-471e-8071-0ffc382d6ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 13:17:59.391970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:59.391929    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-z544g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 13:17:59.392171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:17:59.391993    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" podUID="08586908-beb4-471e-8071-0ffc382d6ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 13:18:00.391546 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:18:00.391514    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-z544g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 13:18:00.391766 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:18:00.391588    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" podUID="08586908-beb4-471e-8071-0ffc382d6ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 13:18:01.392364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:18:01.392323    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-z544g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 13:18:01.392836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:18:01.392399    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g" podUID="08586908-beb4-471e-8071-0ffc382d6ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 13:18:02.939812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:18:02.939779    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:18:02.942222 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:18:02.940112    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:18:02.942222 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:18:02.940955    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-z544g"
Nov 03 13:18:03.411049 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:18:03.410967    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:18:33.411785 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:18:33.411704    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:19:03.411196 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:19:03.411153    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:19:25.910982 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:25.910950    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"]
Nov 03 13:19:25.914880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:25.914855    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:25.917852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:25.917827    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-db71f4-serving-cert\""
Nov 03 13:19:25.924849 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:25.924820    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-metrics-certs\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:25.925091 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:25.925072    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kn29\" (UniqueName: \"kubernetes.io/projected/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-kube-api-access-8kn29\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:25.925289 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:25.925272    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:25.925449 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:25.925430    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:25.928269 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:25.928244    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"]
Nov 03 13:19:26.026954 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.026919    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kn29\" (UniqueName: \"kubernetes.io/projected/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-kube-api-access-8kn29\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:26.027159 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.027016    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:26.027159 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.027039    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:26.027159 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.027084    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-metrics-certs\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:26.027499 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:19:26.027159    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-serving-cert: secret "maas-api-e2e-inf-b-db71f4-serving-cert" not found
Nov 03 13:19:26.027499 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:19:26.027220    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls podName:c1c457a9-8beb-4fca-aaaa-241f3cc4e49c nodeName:}" failed. No retries permitted until 2026-11-03 13:19:26.527201812 +0000 UTC m=+654.338869991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls") pod "maas-api-e2e-inf-b-db71f4-86d869785c-4wn27" (UID: "c1c457a9-8beb-4fca-aaaa-241f3cc4e49c") : secret "maas-api-e2e-inf-b-db71f4-serving-cert" not found
Nov 03 13:19:26.027919 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.027896    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:26.029629 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.029605    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-metrics-certs\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:26.038606 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.038578    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kn29\" (UniqueName: \"kubernetes.io/projected/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-kube-api-access-8kn29\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:26.532301 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.532264    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:26.534755 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.534727    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls\") pod \"maas-api-e2e-inf-b-db71f4-86d869785c-4wn27\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:26.833854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:26.833750    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:27.113655 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:27.113629    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"]
Nov 03 13:19:27.115776 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:19:27.115749    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c457a9_8beb_4fca_aaaa_241f3cc4e49c.slice/crio-4f1694dea1b81a624f5e23ab5107ee3adf38ff9c51b82a02cbbc2835e7838ace WatchSource:0}: Error finding container 4f1694dea1b81a624f5e23ab5107ee3adf38ff9c51b82a02cbbc2835e7838ace: Status 404 returned error can't find the container with id 4f1694dea1b81a624f5e23ab5107ee3adf38ff9c51b82a02cbbc2835e7838ace
Nov 03 13:19:27.117211 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:27.117192    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:19:27.880420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:27.880325    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27" event={"ID":"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c","Type":"ContainerStarted","Data":"191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c"}
Nov 03 13:19:27.880420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:27.880395    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27" event={"ID":"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c","Type":"ContainerStarted","Data":"4f1694dea1b81a624f5e23ab5107ee3adf38ff9c51b82a02cbbc2835e7838ace"}
Nov 03 13:19:27.880633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:27.880431    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:27.901266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:27.901223    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27" podStartSLOduration=2.436582644 podStartE2EDuration="2.901209962s" totalImagesPullingTime="464.627318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:27.898489339 +0000 UTC m=+655.710157554" watchObservedRunningTime="2026-11-03 13:19:27.901209962 +0000 UTC m=+655.712878156"
Nov 03 13:19:33.413770 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:19:33.413725    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:19:33.891704 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:33.891678    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:19:38.656019 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.655977    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"]
Nov 03 13:19:38.660593 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.660558    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.671459 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.671431    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"openshift-service-ca.crt\""
Nov 03 13:19:38.671612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.671442    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"kube-root-ca.crt\""
Nov 03 13:19:38.677585 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.677554    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"]
Nov 03 13:19:38.728393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.724549    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"distinct-w4-c381a7-kserve-self-signed-certs\""
Nov 03 13:19:38.728393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.725281    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"default-dockercfg-d8rbk\""
Nov 03 13:19:38.757380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.755966    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-dshm\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.757380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.756023    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27ae67-e693-4daa-a521-ca104653b406-tls-certs\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.757380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.756139    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-home\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.757380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.756177    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-tmp-dir\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.757380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.756205    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqh7\" (UniqueName: \"kubernetes.io/projected/8a27ae67-e693-4daa-a521-ca104653b406-kube-api-access-8rqh7\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.757380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.756233    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-model-cache\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.857105 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.857062    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-dshm\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.857105 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.857112    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27ae67-e693-4daa-a521-ca104653b406-tls-certs\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.857398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.857207    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-home\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.857398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.857245    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-tmp-dir\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.857398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.857269    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqh7\" (UniqueName: \"kubernetes.io/projected/8a27ae67-e693-4daa-a521-ca104653b406-kube-api-access-8rqh7\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.857398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.857294    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-model-cache\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.857664 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.857640    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-tmp-dir\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.857826 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.857805    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-home\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.858112 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.858087    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-model-cache\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.859673 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.859648    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-dshm\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.860018 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.860002    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27ae67-e693-4daa-a521-ca104653b406-tls-certs\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.867977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.867951    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqh7\" (UniqueName: \"kubernetes.io/projected/8a27ae67-e693-4daa-a521-ca104653b406-kube-api-access-8rqh7\") pod \"distinct-w4-c381a7-kserve-69575dc48-xtbg4\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:38.977952 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:38.977917    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:39.266738 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:39.266713    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"]
Nov 03 13:19:39.268840 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:19:39.268803    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a27ae67_e693_4daa_a521_ca104653b406.slice/crio-1476841beec76a827e717fe9ce733efb680dc39a2d43c4faa7136fa03621c174 WatchSource:0}: Error finding container 1476841beec76a827e717fe9ce733efb680dc39a2d43c4faa7136fa03621c174: Status 404 returned error can't find the container with id 1476841beec76a827e717fe9ce733efb680dc39a2d43c4faa7136fa03621c174
Nov 03 13:19:39.950581 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:39.950511    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" event={"ID":"8a27ae67-e693-4daa-a521-ca104653b406","Type":"ContainerStarted","Data":"1476841beec76a827e717fe9ce733efb680dc39a2d43c4faa7136fa03621c174"}
Nov 03 13:19:40.957648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:40.957613    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" event={"ID":"8a27ae67-e693-4daa-a521-ca104653b406","Type":"ContainerStarted","Data":"14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210"}
Nov 03 13:19:40.976472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:40.976415    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" podStartSLOduration=1.692475619 podStartE2EDuration="2.97639565s" totalImagesPullingTime="1.283920031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:40.97372995 +0000 UTC m=+668.785398140" watchObservedRunningTime="2026-11-03 13:19:40.97639565 +0000 UTC m=+668.788063843"
Nov 03 13:19:40.978817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:40.978804    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:42.965200 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:42.965168    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"]
Nov 03 13:19:42.969488 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:42.969465    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:42.978649 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:42.978621    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-17adbf\"/\"openshift-service-ca.crt\""
Nov 03 13:19:42.978970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:42.978951    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-17adbf\"/\"kube-root-ca.crt\""
Nov 03 13:19:42.979929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:42.979909    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-17adbf\"/\"default-dockercfg-57bcc\""
Nov 03 13:19:42.980011 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:42.979926    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-17adbf\"/\"test-model-17adbf-kserve-self-signed-certs\""
Nov 03 13:19:43.007830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.007800    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"]
Nov 03 13:19:43.099732 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.099686    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-dshm\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.099931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.099743    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tmp-dir\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.099931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.099911    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-model-cache\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.100050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.099985    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-home\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.100050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.100014    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6s4l\" (UniqueName: \"kubernetes.io/projected/4f0e24b4-13ae-451c-9187-78e568c6e1c8-kube-api-access-s6s4l\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.100050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.100039    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tls-certs\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.201413 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.201380    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-dshm\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.201583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.201434    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tmp-dir\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.201583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.201488    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-model-cache\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.201583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.201522    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-home\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.201583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.201548    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6s4l\" (UniqueName: \"kubernetes.io/projected/4f0e24b4-13ae-451c-9187-78e568c6e1c8-kube-api-access-s6s4l\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.201583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.201565    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tls-certs\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.202022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.201999    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-model-cache\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.202104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.202031    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-home\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.202104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.202000    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tmp-dir\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.203781 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.203761    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-dshm\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.204081 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.204059    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tls-certs\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.211967 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.211942    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6s4l\" (UniqueName: \"kubernetes.io/projected/4f0e24b4-13ae-451c-9187-78e568c6e1c8-kube-api-access-s6s4l\") pod \"test-model-17adbf-kserve-b549bbfd9-6kqhk\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") " pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.282137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.282043    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:43.560321 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.560290    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"]
Nov 03 13:19:43.564375 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:19:43.563493    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f0e24b4_13ae_451c_9187_78e568c6e1c8.slice/crio-5d666ba01037a946e0cbcf6e06fed396e91ca6957c38f16ec43cb0097d952503 WatchSource:0}: Error finding container 5d666ba01037a946e0cbcf6e06fed396e91ca6957c38f16ec43cb0097d952503: Status 404 returned error can't find the container with id 5d666ba01037a946e0cbcf6e06fed396e91ca6957c38f16ec43cb0097d952503
Nov 03 13:19:43.977095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.977050    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk" event={"ID":"4f0e24b4-13ae-451c-9187-78e568c6e1c8","Type":"ContainerStarted","Data":"865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751"}
Nov 03 13:19:43.977645 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.977109    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk" event={"ID":"4f0e24b4-13ae-451c-9187-78e568c6e1c8","Type":"ContainerStarted","Data":"5d666ba01037a946e0cbcf6e06fed396e91ca6957c38f16ec43cb0097d952503"}
Nov 03 13:19:43.994758 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:43.994505    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk" podStartSLOduration=1.9944861870000001 podStartE2EDuration="1.994486187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:43.99185025 +0000 UTC m=+671.803518467" watchObservedRunningTime="2026-11-03 13:19:43.994486187 +0000 UTC m=+671.806154376"
Nov 03 13:19:44.282404 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:44.282297    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:48.978647 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:48.978558    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:48.991441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:48.991414    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:49.017060 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:49.017028    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:19:52.215294 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.215253    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"]
Nov 03 13:19:52.235656 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.235624    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"]
Nov 03 13:19:52.235968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.235951    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.239867 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.239843    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"distinct-2-w4-c381a7-kserve-self-signed-certs\""
Nov 03 13:19:52.401535 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.401492    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-model-cache\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.401730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.401544    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875a3afa-0140-40a0-956c-32821edd1f8b-tls-certs\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.401730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.401690    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-dshm\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.401816 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.401744    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-home\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.401856 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.401831    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-tmp-dir\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.401902 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.401874    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwrp\" (UniqueName: \"kubernetes.io/projected/875a3afa-0140-40a0-956c-32821edd1f8b-kube-api-access-wbwrp\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.503436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.503311    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-model-cache\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.503436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.503384    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875a3afa-0140-40a0-956c-32821edd1f8b-tls-certs\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.503683 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.503445    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-dshm\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.503683 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.503487    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-home\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.503683 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.503504    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-tmp-dir\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.503683 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.503525    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwrp\" (UniqueName: \"kubernetes.io/projected/875a3afa-0140-40a0-956c-32821edd1f8b-kube-api-access-wbwrp\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.503918 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.503892    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-model-cache\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.503977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.503916    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-tmp-dir\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.504173 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.504130    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-home\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.506132 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.506099    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-dshm\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.506523 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.506498    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875a3afa-0140-40a0-956c-32821edd1f8b-tls-certs\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.510726 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.510703    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwrp\" (UniqueName: \"kubernetes.io/projected/875a3afa-0140-40a0-956c-32821edd1f8b-kube-api-access-wbwrp\") pod \"distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") " pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.556280 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.556237    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:52.850420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:52.850388    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"]
Nov 03 13:19:52.852057 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:19:52.852024    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod875a3afa_0140_40a0_956c_32821edd1f8b.slice/crio-e610a1038b417136045160b6d57b4f80d0fdf5895c4dfb9c15b6a7837a1c44cc WatchSource:0}: Error finding container e610a1038b417136045160b6d57b4f80d0fdf5895c4dfb9c15b6a7837a1c44cc: Status 404 returned error can't find the container with id e610a1038b417136045160b6d57b4f80d0fdf5895c4dfb9c15b6a7837a1c44cc
Nov 03 13:19:53.033852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:53.033813    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg" event={"ID":"875a3afa-0140-40a0-956c-32821edd1f8b","Type":"ContainerStarted","Data":"d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a"}
Nov 03 13:19:53.034060 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:53.033865    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg" event={"ID":"875a3afa-0140-40a0-956c-32821edd1f8b","Type":"ContainerStarted","Data":"e610a1038b417136045160b6d57b4f80d0fdf5895c4dfb9c15b6a7837a1c44cc"}
Nov 03 13:19:53.050324 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:53.050266    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg" podStartSLOduration=1.050246935 podStartE2EDuration="1.050246935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:53.049025913 +0000 UTC m=+680.860694113" watchObservedRunningTime="2026-11-03 13:19:53.050246935 +0000 UTC m=+680.861915126"
Nov 03 13:19:53.282833 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:53.282799    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:53.296238 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:53.296215    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:19:53.556910 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:53.556827    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:19:54.050566 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:19:54.050534    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:20:01.227489 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.227457    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"]
Nov 03 13:20:01.232049 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.232017    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.242678 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.242648    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e440bc-openshift-default-dockercfg-84hml\""
Nov 03 13:20:01.245391 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.245362    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"]
Nov 03 13:20:01.288626 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.288582    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.288802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.288640    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.288802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.288754    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.288893 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.288802    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.288947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.288921    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.289051 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.289031    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde47a9b-1631-445e-bd1f-98e1da600c0a-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.289121 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.289080    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.289166 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.289121    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5ngl\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-kube-api-access-j5ngl\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.289224 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.289204    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.343260 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.343219    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"]
Nov 03 13:20:01.347144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.347127    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.350221 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.350196    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c381a7-openshift-default-dockercfg-c5nmd\""
Nov 03 13:20:01.358842 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.358811    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"]
Nov 03 13:20:01.389933 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.389888    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.390107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.389944    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390019    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.390107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390078    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390177    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390212    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.390315 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390257    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390315 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390292    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390536    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390596    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390644 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390621    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.390686 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390663    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390754    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.390860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390801    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.390860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390834    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpzzw\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-kube-api-access-jpzzw\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.390991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390886    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390907    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde47a9b-1631-445e-bd1f-98e1da600c0a-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.390991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390937    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.390991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390966    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.391199 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.390996    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5ngl\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-kube-api-access-j5ngl\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.391199 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.391019    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c8ebc5-ed58-43e0-96ab-70c7da119904-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.391199 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.391071    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.391667 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.391645    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde47a9b-1631-445e-bd1f-98e1da600c0a-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.393746 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.393724    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.394798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.394778    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.399282 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.399257    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.400548 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.400531    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5ngl\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-kube-api-access-j5ngl\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.491840 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.491752    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c8ebc5-ed58-43e0-96ab-70c7da119904-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.491840 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.491796    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492043 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.491840    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492043 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.491895    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492043 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.491952    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492043 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.491997    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492043 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.492031    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.492071    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpzzw\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-kube-api-access-jpzzw\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.492107    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492337 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.492312    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492536 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.492513    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492536 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.492523    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.492599    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.492698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.492634    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c8ebc5-ed58-43e0-96ab-70c7da119904-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.494499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.494476    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.494604 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.494541    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.509297 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.509262    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpzzw\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-kube-api-access-jpzzw\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.509677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.509655    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.549719 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.549677    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:01.662908 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.662874    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:01.832450 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.832425    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"]
Nov 03 13:20:01.835617 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:20:01.835589    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde47a9b_1631_445e_bd1f_98e1da600c0a.slice/crio-ee676bb9996fc4294c1caa4fba60fe295bfd4452cddd7042b02a99147cf07e4b WatchSource:0}: Error finding container ee676bb9996fc4294c1caa4fba60fe295bfd4452cddd7042b02a99147cf07e4b: Status 404 returned error can't find the container with id ee676bb9996fc4294c1caa4fba60fe295bfd4452cddd7042b02a99147cf07e4b
Nov 03 13:20:01.838689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.838656    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:01.838796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.838733    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:01.838796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.838781    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:01.946676 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.946390    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"]
Nov 03 13:20:01.948722 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:20:01.948696    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c8ebc5_ed58_43e0_96ab_70c7da119904.slice/crio-1c7ecc99b99dc4e4fcbf5ad21462a4f0dee9fa17c90d9147f0b1e2d81ba04fdc WatchSource:0}: Error finding container 1c7ecc99b99dc4e4fcbf5ad21462a4f0dee9fa17c90d9147f0b1e2d81ba04fdc: Status 404 returned error can't find the container with id 1c7ecc99b99dc4e4fcbf5ad21462a4f0dee9fa17c90d9147f0b1e2d81ba04fdc
Nov 03 13:20:01.951878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.951846    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:01.951994 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.951908    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:01.951994 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:01.951935    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:02.089271 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.089181    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" event={"ID":"57c8ebc5-ed58-43e0-96ab-70c7da119904","Type":"ContainerStarted","Data":"e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a"}
Nov 03 13:20:02.089271 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.089242    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" event={"ID":"57c8ebc5-ed58-43e0-96ab-70c7da119904","Type":"ContainerStarted","Data":"1c7ecc99b99dc4e4fcbf5ad21462a4f0dee9fa17c90d9147f0b1e2d81ba04fdc"}
Nov 03 13:20:02.091750 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.091669    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" event={"ID":"dde47a9b-1631-445e-bd1f-98e1da600c0a","Type":"ContainerStarted","Data":"2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272"}
Nov 03 13:20:02.091750 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.091722    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" event={"ID":"dde47a9b-1631-445e-bd1f-98e1da600c0a","Type":"ContainerStarted","Data":"ee676bb9996fc4294c1caa4fba60fe295bfd4452cddd7042b02a99147cf07e4b"}
Nov 03 13:20:02.109752 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.109700    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" podStartSLOduration=1.109686073 podStartE2EDuration="1.109686073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:02.10593652 +0000 UTC m=+689.917604728" watchObservedRunningTime="2026-11-03 13:20:02.109686073 +0000 UTC m=+689.921354253"
Nov 03 13:20:02.125389 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.125178    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" podStartSLOduration=1.125159574 podStartE2EDuration="1.125159574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:02.124889732 +0000 UTC m=+689.936557950" watchObservedRunningTime="2026-11-03 13:20:02.125159574 +0000 UTC m=+689.936827765"
Nov 03 13:20:02.557367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.556797    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:20:02.561376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.557901    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:02.569377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.568484    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 13:20:02.569377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.568570    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 13:20:02.604687 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.604656    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:20:02.682486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:02.682444    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:03.110658 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:03.110613    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:20:03.423810 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:03.423716    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:20:03.553368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:03.551501    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 13:20:03.553368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:03.551564    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 13:20:03.684368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:03.684075    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 13:20:03.684368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:03.684139    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 13:20:04.554826 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:04.554784    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 13:20:04.555127 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:04.555091    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 13:20:04.665371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:04.663795    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 13:20:04.665371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:04.663863    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 13:20:05.552804 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:05.552759    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 13:20:05.553464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:05.553430    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 13:20:05.663795 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:05.663754    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 13:20:05.664046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:05.664018    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 13:20:06.554368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:06.553994    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 13:20:06.554368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:06.554057    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 13:20:06.664364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:06.663790    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 13:20:06.664364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:06.663854    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 13:20:07.552273 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:07.551814    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 13:20:07.552273 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:07.551881    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 13:20:07.664779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:07.664741    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 13:20:07.665417 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:07.665384    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 13:20:08.663603 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:08.663561    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 13:20:08.663768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:08.663631    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 13:20:08.791596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:08.791566    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:08.792002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:08.791959    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:08.792912 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:08.792893    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:20:09.668627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:09.668592    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:09.669040 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:09.669000    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:09.669942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:09.669923    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:20:16.450170 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.450135    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"]
Nov 03 13:20:16.456969 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.456946    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.463257 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.463236    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a4ab8e-openshift-default-dockercfg-dm758\""
Nov 03 13:20:16.469485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.469459    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"]
Nov 03 13:20:16.613083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.613038    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.613083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.613080    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.613292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.613119    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.613292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.613176    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f16b044-d645-494f-a857-00264f18ce26-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.613292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.613232    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.613292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.613266    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.613430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.613337    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.613430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.613393    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f16b044-d645-494f-a857-00264f18ce26-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.613492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.613470    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862fb\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-kube-api-access-862fb\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.714885 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.714803    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862fb\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-kube-api-access-862fb\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.714885 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.714866    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.714888    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.714930    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.714947    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f16b044-d645-494f-a857-00264f18ce26-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.714971    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.715390    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715520 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.715436    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715520 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.715469    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715520 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.715392    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715688 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.715532    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f16b044-d645-494f-a857-00264f18ce26-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715688 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.715590    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715688 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.715668    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.715868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.715837    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f16b044-d645-494f-a857-00264f18ce26-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.717315 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.717290    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.717929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.717908    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f16b044-d645-494f-a857-00264f18ce26-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.723173 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.723148    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.723383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.723361    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862fb\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-kube-api-access-862fb\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:16.771912 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:16.771877    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:17.069944 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:17.069917    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"]
Nov 03 13:20:17.071611 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:20:17.071582    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f16b044_d645_494f_a857_00264f18ce26.slice/crio-6d86c4b86bade7c4e23206690b6ae4cd80278ce56b4e4a97d018dab3b44a42bd WatchSource:0}: Error finding container 6d86c4b86bade7c4e23206690b6ae4cd80278ce56b4e4a97d018dab3b44a42bd: Status 404 returned error can't find the container with id 6d86c4b86bade7c4e23206690b6ae4cd80278ce56b4e4a97d018dab3b44a42bd
Nov 03 13:20:17.074180 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:17.074152    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:17.074269 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:17.074215    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:17.074269 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:17.074253    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:17.194914 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:17.194857    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" event={"ID":"9f16b044-d645-494f-a857-00264f18ce26","Type":"ContainerStarted","Data":"857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69"}
Nov 03 13:20:17.194914 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:17.194919    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" event={"ID":"9f16b044-d645-494f-a857-00264f18ce26","Type":"ContainerStarted","Data":"6d86c4b86bade7c4e23206690b6ae4cd80278ce56b4e4a97d018dab3b44a42bd"}
Nov 03 13:20:17.215563 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:17.215502    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" podStartSLOduration=1.215483304 podStartE2EDuration="1.215483304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:17.214585436 +0000 UTC m=+705.026253648" watchObservedRunningTime="2026-11-03 13:20:17.215483304 +0000 UTC m=+705.027151494"
Nov 03 13:20:17.772647 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:17.772611    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:18.774642 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:18.774609    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 13:20:18.775028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:18.774666    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 13:20:19.772308 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:19.772269    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 13:20:19.772492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:19.772329    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 13:20:20.772868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:20.772829    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 13:20:20.773371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:20.772888    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 13:20:21.773530 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:21.773488    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 13:20:21.774027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:21.773545    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 13:20:22.772875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:22.772834    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 13:20:22.773066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:22.772899    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 13:20:24.481603 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:24.481544    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:24.482184 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:24.482086    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:24.482848 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:24.482827    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:20:29.300453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:29.300391    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-17b551"]
Nov 03 13:20:29.304411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:29.304392    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:29.308779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:29.308752    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncr7v\""
Nov 03 13:20:29.309401 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:29.309372    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-17b551"]
Nov 03 13:20:29.385815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:29.385780    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z597\" (UniqueName: \"kubernetes.io/projected/1cb5b352-abe3-452e-b1c8-549d2346c384-kube-api-access-9z597\") pod \"test-curl-7092-17b551\" (UID: \"1cb5b352-abe3-452e-b1c8-549d2346c384\") " pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:29.486547 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:29.486508    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z597\" (UniqueName: \"kubernetes.io/projected/1cb5b352-abe3-452e-b1c8-549d2346c384-kube-api-access-9z597\") pod \"test-curl-7092-17b551\" (UID: \"1cb5b352-abe3-452e-b1c8-549d2346c384\") " pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:29.495718 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:29.495690    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z597\" (UniqueName: \"kubernetes.io/projected/1cb5b352-abe3-452e-b1c8-549d2346c384-kube-api-access-9z597\") pod \"test-curl-7092-17b551\" (UID: \"1cb5b352-abe3-452e-b1c8-549d2346c384\") " pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:29.620413 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:29.620323    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:29.893124 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:29.893100    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-17b551"]
Nov 03 13:20:29.894958 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:20:29.894929    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cb5b352_abe3_452e_b1c8_549d2346c384.slice/crio-4e0507987d1bacd95721a9afbff46a90710864632ae7c907b7ea1f4b20a9ab7b WatchSource:0}: Error finding container 4e0507987d1bacd95721a9afbff46a90710864632ae7c907b7ea1f4b20a9ab7b: Status 404 returned error can't find the container with id 4e0507987d1bacd95721a9afbff46a90710864632ae7c907b7ea1f4b20a9ab7b
Nov 03 13:20:30.278525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:30.278480    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-17b551" event={"ID":"1cb5b352-abe3-452e-b1c8-549d2346c384","Type":"ContainerStarted","Data":"4e0507987d1bacd95721a9afbff46a90710864632ae7c907b7ea1f4b20a9ab7b"}
Nov 03 13:20:31.409757 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.409728    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"]
Nov 03 13:20:31.415213 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.415191    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.427696 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.427622    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"]
Nov 03 13:20:31.508946 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.508911    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f6efa-9355-46af-9a6a-9f857a1dedb6-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.509113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.508967    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.509113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.509072    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.509113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.509110    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.509218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.509134    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.509289 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.509270    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.509362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.509331    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz668\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-kube-api-access-lz668\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.509415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.509400    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.509462 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.509448    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610318    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f6efa-9355-46af-9a6a-9f857a1dedb6-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610388    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610419    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610444    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610476    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610524    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610579    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz668\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-kube-api-access-lz668\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610607    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610893 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610632    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610914    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.610948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.610934    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.611088 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.611038    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.611088 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.611078    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.611460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.611442    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f6efa-9355-46af-9a6a-9f857a1dedb6-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.612999 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.612980    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.613094 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.613006    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.617676 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.617651    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.617676 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.617668    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz668\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-kube-api-access-lz668\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:31.732214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:31.732179    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:32.014329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.014294    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"]
Nov 03 13:20:32.017373 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:20:32.017328    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195f6efa_9355_46af_9a6a_9f857a1dedb6.slice/crio-e57910383057b6beb27a57d9b6f117122f43c97b9d93c5fd40d082e29241d217 WatchSource:0}: Error finding container e57910383057b6beb27a57d9b6f117122f43c97b9d93c5fd40d082e29241d217: Status 404 returned error can't find the container with id e57910383057b6beb27a57d9b6f117122f43c97b9d93c5fd40d082e29241d217
Nov 03 13:20:32.020176 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.020146    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:32.020244 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.020214    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:32.020281 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.020241    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:32.294604 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.294522    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-17b551" event={"ID":"1cb5b352-abe3-452e-b1c8-549d2346c384","Type":"ContainerStarted","Data":"097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30"}
Nov 03 13:20:32.300876 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.300834    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" event={"ID":"195f6efa-9355-46af-9a6a-9f857a1dedb6","Type":"ContainerStarted","Data":"71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf"}
Nov 03 13:20:32.301068 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.300892    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" event={"ID":"195f6efa-9355-46af-9a6a-9f857a1dedb6","Type":"ContainerStarted","Data":"e57910383057b6beb27a57d9b6f117122f43c97b9d93c5fd40d082e29241d217"}
Nov 03 13:20:32.311167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.311114    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7092-17b551" podStartSLOduration=1.8798057959999999 podStartE2EDuration="3.311101422s" totalImagesPullingTime="1.431295626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:32.309171966 +0000 UTC m=+720.120840133" watchObservedRunningTime="2026-11-03 13:20:32.311101422 +0000 UTC m=+720.122769610"
Nov 03 13:20:32.335261 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.335204    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" podStartSLOduration=1.335186649 podStartE2EDuration="1.335186649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:32.32507293 +0000 UTC m=+720.136741149" watchObservedRunningTime="2026-11-03 13:20:32.335186649 +0000 UTC m=+720.146854839"
Nov 03 13:20:32.732404 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:32.732374    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:33.064498 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.064413    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-17b551"]
Nov 03 13:20:33.068135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.068094    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-17b551"]
Nov 03 13:20:33.201632 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.201598    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-78af94"]
Nov 03 13:20:33.202088 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.202075    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1cb5b352-abe3-452e-b1c8-549d2346c384" containerName="test-curl-7092-17b551"
Nov 03 13:20:33.202131 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.202089    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb5b352-abe3-452e-b1c8-549d2346c384" containerName="test-curl-7092-17b551"
Nov 03 13:20:33.202207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.202198    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb5b352-abe3-452e-b1c8-549d2346c384" containerName="test-curl-7092-17b551"
Nov 03 13:20:33.206715 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.206692    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:33.211494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.211465    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-78af94"]
Nov 03 13:20:33.328704 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.328611    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj9pf\" (UniqueName: \"kubernetes.io/projected/a8982ab5-2308-45f9-a136-33f6d4704cb3-kube-api-access-jj9pf\") pod \"test-curl-7092-78af94\" (UID: \"a8982ab5-2308-45f9-a136-33f6d4704cb3\") " pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:33.412127 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:33.412086    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:20:33.429788 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.429754    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj9pf\" (UniqueName: \"kubernetes.io/projected/a8982ab5-2308-45f9-a136-33f6d4704cb3-kube-api-access-jj9pf\") pod \"test-curl-7092-78af94\" (UID: \"a8982ab5-2308-45f9-a136-33f6d4704cb3\") " pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:33.464853 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.464818    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj9pf\" (UniqueName: \"kubernetes.io/projected/a8982ab5-2308-45f9-a136-33f6d4704cb3-kube-api-access-jj9pf\") pod \"test-curl-7092-78af94\" (UID: \"a8982ab5-2308-45f9-a136-33f6d4704cb3\") " pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:33.523507 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.523471    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:33.734206 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.734171    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 13:20:33.734673 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.734232    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 13:20:33.828254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:33.828228    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-78af94"]
Nov 03 13:20:33.829883 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:20:33.829856    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8982ab5_2308_45f9_a136_33f6d4704cb3.slice/crio-e5128026d0f520019cecd2b0e70bd9e2ce71a50f6155c5da6a2c856092b6418a WatchSource:0}: Error finding container e5128026d0f520019cecd2b0e70bd9e2ce71a50f6155c5da6a2c856092b6418a: Status 404 returned error can't find the container with id e5128026d0f520019cecd2b0e70bd9e2ce71a50f6155c5da6a2c856092b6418a
Nov 03 13:20:34.314021 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:34.313977    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7092-17b551" podUID="1cb5b352-abe3-452e-b1c8-549d2346c384" containerName="test-curl-7092-17b551" containerID="cri-o://097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30" gracePeriod=2
Nov 03 13:20:34.316440 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:34.316397    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-17b551\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="1cb5b352-abe3-452e-b1c8-549d2346c384" pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:34.318114 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:34.318087    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-17b551\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="1cb5b352-abe3-452e-b1c8-549d2346c384" pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:34.318557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:34.318536    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-78af94" event={"ID":"a8982ab5-2308-45f9-a136-33f6d4704cb3","Type":"ContainerStarted","Data":"c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1"}
Nov 03 13:20:34.318611 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:34.318571    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-78af94" event={"ID":"a8982ab5-2308-45f9-a136-33f6d4704cb3","Type":"ContainerStarted","Data":"e5128026d0f520019cecd2b0e70bd9e2ce71a50f6155c5da6a2c856092b6418a"}
Nov 03 13:20:34.332223 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:34.332104    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7092-78af94" podStartSLOduration=1.332092235 podStartE2EDuration="1.332092235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:34.331639465 +0000 UTC m=+722.143307672" watchObservedRunningTime="2026-11-03 13:20:34.332092235 +0000 UTC m=+722.143760423"
Nov 03 13:20:34.733463 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:34.733429    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 13:20:34.733671 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:34.733492    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 13:20:35.030405 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:35.030314    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-78af94"]
Nov 03 13:20:35.066472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:35.066435    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-78af94"]
Nov 03 13:20:35.733559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:35.733523    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 13:20:35.733738 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:35.733582    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 13:20:36.325929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:36.325869    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7092-78af94" podUID="a8982ab5-2308-45f9-a136-33f6d4704cb3" containerName="test-curl-7092-78af94" containerID="cri-o://c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1" gracePeriod=2
Nov 03 13:20:36.608098 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:36.608072    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:36.693844 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:36.693800    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb5b352-abe3-452e-b1c8-549d2346c384-kube-api-access-9z597\" (UniqueName: \"kubernetes.io/projected/1cb5b352-abe3-452e-b1c8-549d2346c384-kube-api-access-9z597\") pod \"1cb5b352-abe3-452e-b1c8-549d2346c384\" (UID: \"1cb5b352-abe3-452e-b1c8-549d2346c384\") "
Nov 03 13:20:36.696569 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:36.696539    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb5b352-abe3-452e-b1c8-549d2346c384-kube-api-access-9z597" pod "1cb5b352-abe3-452e-b1c8-549d2346c384" (UID: "1cb5b352-abe3-452e-b1c8-549d2346c384"). InnerVolumeSpecName "kube-api-access-9z597". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:36.733243 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:36.733207    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 13:20:36.733449 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:36.733268    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 13:20:36.794880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:36.794843    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z597\" (UniqueName: \"kubernetes.io/projected/1cb5b352-abe3-452e-b1c8-549d2346c384-kube-api-access-9z597\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:37.350370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:37.349635    4447 generic.go:356] "Generic (PLEG): container finished" podID="1cb5b352-abe3-452e-b1c8-549d2346c384" containerID="097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30" exitCode=137
Nov 03 13:20:37.350370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:37.349726    4447 scope.go:122] "RemoveContainer" containerID="097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30"
Nov 03 13:20:37.350370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:37.349886    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:37.391888 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:37.391853    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-17b551\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="1cb5b352-abe3-452e-b1c8-549d2346c384" pod="opendatahub/test-curl-7092-17b551"
Nov 03 13:20:37.392262 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:37.392244    4447 scope.go:122] "RemoveContainer" containerID="097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30"
Nov 03 13:20:37.393484 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:37.393448    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30\": container with ID starting with 097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30 not found: ID does not exist" containerID="097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30"
Nov 03 13:20:37.393610 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:37.393488    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30"} err="failed to get container status \"097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30\": rpc error: code = NotFound desc = could not find container \"097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30\": container with ID starting with 097ce47a9c54aeb66e40a653c81e8a76d89b4e8ef2c66ef1741477b8e73a7d30 not found: ID does not exist"
Nov 03 13:20:37.733035 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:37.732999    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 13:20:37.733211 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:37.733057    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 13:20:38.320789 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:38.320757    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb5b352-abe3-452e-b1c8-549d2346c384" path="/var/lib/kubelet/pods/1cb5b352-abe3-452e-b1c8-549d2346c384/volumes"
Nov 03 13:20:38.508733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:38.508682    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:38.623236 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:38.623133    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8982ab5-2308-45f9-a136-33f6d4704cb3-kube-api-access-jj9pf\" (UniqueName: \"kubernetes.io/projected/a8982ab5-2308-45f9-a136-33f6d4704cb3-kube-api-access-jj9pf\") pod \"a8982ab5-2308-45f9-a136-33f6d4704cb3\" (UID: \"a8982ab5-2308-45f9-a136-33f6d4704cb3\") "
Nov 03 13:20:38.625262 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:38.625233    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8982ab5-2308-45f9-a136-33f6d4704cb3-kube-api-access-jj9pf" pod "a8982ab5-2308-45f9-a136-33f6d4704cb3" (UID: "a8982ab5-2308-45f9-a136-33f6d4704cb3"). InnerVolumeSpecName "kube-api-access-jj9pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:38.724429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:38.724391    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj9pf\" (UniqueName: \"kubernetes.io/projected/a8982ab5-2308-45f9-a136-33f6d4704cb3-kube-api-access-jj9pf\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:38.733324 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:38.733285    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 13:20:38.733495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:38.733339    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 13:20:39.365817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:39.365773    4447 generic.go:356] "Generic (PLEG): container finished" podID="a8982ab5-2308-45f9-a136-33f6d4704cb3" containerID="c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1" exitCode=137
Nov 03 13:20:39.366002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:39.365819    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:39.366002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:39.365870    4447 scope.go:122] "RemoveContainer" containerID="c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1"
Nov 03 13:20:39.379508 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:39.376543    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-78af94\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="a8982ab5-2308-45f9-a136-33f6d4704cb3" pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:39.383082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:39.382907    4447 scope.go:122] "RemoveContainer" containerID="c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1"
Nov 03 13:20:39.383280 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:39.383258    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1\": container with ID starting with c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1 not found: ID does not exist" containerID="c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1"
Nov 03 13:20:39.383321 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:39.383294    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1"} err="failed to get container status \"c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1\": rpc error: code = NotFound desc = could not find container \"c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1\": container with ID starting with c2605b6cf3a63f7d45bed722d54deced071bd371a148e54442e863c1ddc858f1 not found: ID does not exist"
Nov 03 13:20:39.738675 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:39.738642    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:39.739128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:39.738974    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:39.740021 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:39.739980    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:20:39.740580 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:39.740554    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-78af94\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="a8982ab5-2308-45f9-a136-33f6d4704cb3" pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:39.760736 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:20:39.760702    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-78af94\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="a8982ab5-2308-45f9-a136-33f6d4704cb3" pod="opendatahub/test-curl-7092-78af94"
Nov 03 13:20:40.319484 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:40.319451    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8982ab5-2308-45f9-a136-33f6d4704cb3" path="/var/lib/kubelet/pods/a8982ab5-2308-45f9-a136-33f6d4704cb3/volumes"
Nov 03 13:20:46.356799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.354297    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"]
Nov 03 13:20:46.356799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.355092    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a8982ab5-2308-45f9-a136-33f6d4704cb3" containerName="test-curl-7092-78af94"
Nov 03 13:20:46.356799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.355112    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8982ab5-2308-45f9-a136-33f6d4704cb3" containerName="test-curl-7092-78af94"
Nov 03 13:20:46.356799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.355281    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8982ab5-2308-45f9-a136-33f6d4704cb3" containerName="test-curl-7092-78af94"
Nov 03 13:20:46.509365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.486553    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"]
Nov 03 13:20:46.509365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.486694    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.509365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.496765    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"]
Nov 03 13:20:46.534367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.529961    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.534367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.530025    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.534367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.530062    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wnl\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-kube-api-access-t7wnl\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.534367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.530089    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.534367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.530136    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.534367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.530199    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.534367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.530223    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6043f9d-679a-490b-8753-b32e618cce8c-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.534367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.530305    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.534367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.530336    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6043f9d-679a-490b-8753-b32e618cce8c-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.561367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.560635    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633030    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633086    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633120    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633151    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633192    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633229    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wnl\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-kube-api-access-t7wnl\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633254    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633280    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633312    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633382    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633418    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633492    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pnrp\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-kube-api-access-5pnrp\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633520    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633547    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6043f9d-679a-490b-8753-b32e618cce8c-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633606    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad2c8ea-c4be-479f-a357-bfa000088cda-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633663    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633693    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6043f9d-679a-490b-8753-b32e618cce8c-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.633741    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.634726    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.657425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.635236    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.679083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.670547    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.679083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.670893    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.697004 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.688828    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6043f9d-679a-490b-8753-b32e618cce8c-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.697004 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.689166    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.697004 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.689854    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6043f9d-679a-490b-8753-b32e618cce8c-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.697004 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.690774    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.719489 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.719461    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wnl\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-kube-api-access-t7wnl\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.722845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.722800    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"]
Nov 03 13:20:46.755836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.755799    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnrp\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-kube-api-access-5pnrp\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.756756 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.756730    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad2c8ea-c4be-479f-a357-bfa000088cda-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.756972 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.756943    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.757173 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.757155    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.757319 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.757300    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.757488 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.757471    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.757660 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.757635    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.757797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.757780    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.757991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.757972    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.759293 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.759272    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.760253 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.760232    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad2c8ea-c4be-479f-a357-bfa000088cda-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.760762 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.760741    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.761238 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.761221    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.761964 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.761937    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.784194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.784121    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnrp\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-kube-api-access-5pnrp\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.786592 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.786544    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.790067 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.789903    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:46.820445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.819846    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.820850 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.820826    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:46.823314 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:46.823280    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:47.153559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.153443    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"]
Nov 03 13:20:47.166990 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.166957    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"]
Nov 03 13:20:47.171498 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:20:47.171456    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6043f9d_679a_490b_8753_b32e618cce8c.slice/crio-1b1371ba102d2a47e8e15360772a263dc34b5cc7ce0f6545e6b269d05288e1c9 WatchSource:0}: Error finding container 1b1371ba102d2a47e8e15360772a263dc34b5cc7ce0f6545e6b269d05288e1c9: Status 404 returned error can't find the container with id 1b1371ba102d2a47e8e15360772a263dc34b5cc7ce0f6545e6b269d05288e1c9
Nov 03 13:20:47.174803 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.174764    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:47.174896 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.174873    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:47.174955 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.174941    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:47.175486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.175454    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:47.175665 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.175643    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:47.175817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.175795    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:20:47.511136 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.511096    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" event={"ID":"8ad2c8ea-c4be-479f-a357-bfa000088cda","Type":"ContainerStarted","Data":"292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13"}
Nov 03 13:20:47.511659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.511636    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" event={"ID":"8ad2c8ea-c4be-479f-a357-bfa000088cda","Type":"ContainerStarted","Data":"dec790faf3537cea4576cd16759db86d868e1e1119d290ad2fa7997fcc69f701"}
Nov 03 13:20:47.515183 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.515130    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" event={"ID":"b6043f9d-679a-490b-8753-b32e618cce8c","Type":"ContainerStarted","Data":"cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68"}
Nov 03 13:20:47.515183 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.515176    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" event={"ID":"b6043f9d-679a-490b-8753-b32e618cce8c","Type":"ContainerStarted","Data":"1b1371ba102d2a47e8e15360772a263dc34b5cc7ce0f6545e6b269d05288e1c9"}
Nov 03 13:20:47.529880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.529820    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" podStartSLOduration=1.529803758 podStartE2EDuration="1.529803758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:47.528158791 +0000 UTC m=+735.339827033" watchObservedRunningTime="2026-11-03 13:20:47.529803758 +0000 UTC m=+735.341471946"
Nov 03 13:20:47.546890 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.546835    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" podStartSLOduration=1.546822046 podStartE2EDuration="1.546822046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:47.54538286 +0000 UTC m=+735.357051050" watchObservedRunningTime="2026-11-03 13:20:47.546822046 +0000 UTC m=+735.358490233"
Nov 03 13:20:47.791129 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.791037    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:47.824128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:47.824097    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:48.792507 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:48.792472    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:48.792888 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:48.792533    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 13:20:48.825849 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:48.825816    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 13:20:48.825998 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:48.825869    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 13:20:49.790755 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:49.790716    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:49.790943 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:49.790784    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 13:20:49.824809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:49.824771    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 13:20:49.825214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:49.824821    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 13:20:50.790797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:50.790757    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:50.791112 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:50.791073    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 13:20:50.824448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:50.824412    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 13:20:50.824616 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:50.824508    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 13:20:51.791396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:51.790798    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:51.791396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:51.790868    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 13:20:51.824744 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:51.824707    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 13:20:51.825034 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:51.825000    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 13:20:52.791413 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:52.791376    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:52.791794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:52.791443    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 13:20:52.823977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:52.823940    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 13:20:52.824149 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:52.823992    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 13:20:53.790873 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:53.790829    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:53.791052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:53.790913    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 13:20:53.824112 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:53.824072    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 13:20:53.824530 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:53.824141    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 13:20:54.795185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:54.795151    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:54.795475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:54.795444    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:54.796223 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:54.796207    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:20:54.828542 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:54.828517    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:54.828911 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:54.828821    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:20:54.829607 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:20:54.829583    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:21:01.416837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.416803    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"]
Nov 03 13:21:01.422209 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.422185    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.437194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.437164    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"]
Nov 03 13:21:01.563384 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.563318    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.563527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.563399    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.563527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.563481    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a9701a-c232-4626-938e-606c4ebfc5db-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.563527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.563514    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.563641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.563610    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.563675 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.563651    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xfn2\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-kube-api-access-9xfn2\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.563717 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.563693    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.563766 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.563724    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.563834 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.563811    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a9701a-c232-4626-938e-606c4ebfc5db-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.664834 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.664803    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.664997 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.664845    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xfn2\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-kube-api-access-9xfn2\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.664997 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.664931    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.665144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.665104    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.665201 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.665160    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a9701a-c232-4626-938e-606c4ebfc5db-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.665255 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.665234    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.665310 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.665262    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.665310 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.665286    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.665452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.665403    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a9701a-c232-4626-938e-606c4ebfc5db-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.665452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.665445    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.665612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.665288    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.665807 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.665774    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.668433 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.666028    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a9701a-c232-4626-938e-606c4ebfc5db-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.668433 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.666150    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.668433 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.667646    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.668433 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.667878    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a9701a-c232-4626-938e-606c4ebfc5db-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.672568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.672542    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.672689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.672670    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xfn2\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-kube-api-access-9xfn2\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:01.740178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:01.740131    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:02.025106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:02.025075    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"]
Nov 03 13:21:02.026905 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:21:02.026874    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a9701a_c232_4626_938e_606c4ebfc5db.slice/crio-e2822180ebb17fd63ef56d52d7416bb2bfd7696902ddad140b189eb03a6f6ad4 WatchSource:0}: Error finding container e2822180ebb17fd63ef56d52d7416bb2bfd7696902ddad140b189eb03a6f6ad4: Status 404 returned error can't find the container with id e2822180ebb17fd63ef56d52d7416bb2bfd7696902ddad140b189eb03a6f6ad4
Nov 03 13:21:02.029779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:02.029643    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:02.029779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:02.029728    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:02.029928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:02.029778    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:02.667549 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:02.667513    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" event={"ID":"67a9701a-c232-4626-938e-606c4ebfc5db","Type":"ContainerStarted","Data":"16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f"}
Nov 03 13:21:02.668030 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:02.667571    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" event={"ID":"67a9701a-c232-4626-938e-606c4ebfc5db","Type":"ContainerStarted","Data":"e2822180ebb17fd63ef56d52d7416bb2bfd7696902ddad140b189eb03a6f6ad4"}
Nov 03 13:21:02.686710 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:02.686626    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" podStartSLOduration=1.686610033 podStartE2EDuration="1.686610033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:02.685089047 +0000 UTC m=+750.496757254" watchObservedRunningTime="2026-11-03 13:21:02.686610033 +0000 UTC m=+750.498278222"
Nov 03 13:21:02.754367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:02.753430    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:03.413478 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:21:03.413438    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:21:03.747544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:03.747506    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 13:21:03.748257 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:03.748222    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 13:21:04.160998 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.160899    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"]
Nov 03 13:21:04.166690 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.166665    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.168466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.168438    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"]
Nov 03 13:21:04.169465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.169429    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:21:04.169914 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.169894    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:21:04.194252 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.194221    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpd4c\" (UniqueName: \"kubernetes.io/projected/b3271d72-1127-4a45-bff1-23b7c80b9c86-kube-api-access-zpd4c\") pod \"maas-api-revoke-sub-6540d27d-8ccj8\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.194458 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.194395    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6540d27d-8ccj8\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.194535 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.194511    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6540d27d-8ccj8\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.296007 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.295969    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6540d27d-8ccj8\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.296194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.296038    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6540d27d-8ccj8\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.296194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.296115    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpd4c\" (UniqueName: \"kubernetes.io/projected/b3271d72-1127-4a45-bff1-23b7c80b9c86-kube-api-access-zpd4c\") pod \"maas-api-revoke-sub-6540d27d-8ccj8\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.296696 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.296671    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6540d27d-8ccj8\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.299006 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.298974    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6540d27d-8ccj8\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.304378 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.304317    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpd4c\" (UniqueName: \"kubernetes.io/projected/b3271d72-1127-4a45-bff1-23b7c80b9c86-kube-api-access-zpd4c\") pod \"maas-api-revoke-sub-6540d27d-8ccj8\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.483905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.483869    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:04.761138 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.760865    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 13:21:04.761138 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.760952    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 13:21:04.787147 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:04.787124    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"]
Nov 03 13:21:04.789464 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:21:04.789425    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3271d72_1127_4a45_bff1_23b7c80b9c86.slice/crio-1a9fcd7671ba314b26d2f9e3ceef544bc20f16d9f2ecd2835e52000864216578 WatchSource:0}: Error finding container 1a9fcd7671ba314b26d2f9e3ceef544bc20f16d9f2ecd2835e52000864216578: Status 404 returned error can't find the container with id 1a9fcd7671ba314b26d2f9e3ceef544bc20f16d9f2ecd2835e52000864216578
Nov 03 13:21:05.691483 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:05.691444    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8" event={"ID":"b3271d72-1127-4a45-bff1-23b7c80b9c86","Type":"ContainerStarted","Data":"1a9fcd7671ba314b26d2f9e3ceef544bc20f16d9f2ecd2835e52000864216578"}
Nov 03 13:21:05.742287 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:05.741715    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 13:21:05.742287 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:05.741788    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 13:21:06.764725 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:06.762774    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 13:21:06.764725 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:06.762838    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 13:21:07.752368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:07.752031    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 13:21:07.752368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:07.752093    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 13:21:08.742375 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:08.740926    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 13:21:08.742375 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:08.740988    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 13:21:09.922249 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:09.920674    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:09.922249 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:09.922090    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:09.931002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:09.930981    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:21:11.785935 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:11.785846    4447 generic.go:356] "Generic (PLEG): container finished" podID="b3271d72-1127-4a45-bff1-23b7c80b9c86" containerID="361817f90c88b905e652ed90ea1ddfd04d6199c534225189172f3509af57bd0d" exitCode=0
Nov 03 13:21:11.786302 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:11.785928    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8" event={"ID":"b3271d72-1127-4a45-bff1-23b7c80b9c86","Type":"ContainerDied","Data":"361817f90c88b905e652ed90ea1ddfd04d6199c534225189172f3509af57bd0d"}
Nov 03 13:21:12.971421 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:12.971361    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:13.061590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.061493    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3271d72-1127-4a45-bff1-23b7c80b9c86-kube-api-access-zpd4c\" (UniqueName: \"kubernetes.io/projected/b3271d72-1127-4a45-bff1-23b7c80b9c86-kube-api-access-zpd4c\") pod \"b3271d72-1127-4a45-bff1-23b7c80b9c86\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") "
Nov 03 13:21:13.061769 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.061645    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-xks-ca\") pod \"b3271d72-1127-4a45-bff1-23b7c80b9c86\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") "
Nov 03 13:21:13.061769 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.061675    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-openshift-ca\") pod \"b3271d72-1127-4a45-bff1-23b7c80b9c86\" (UID: \"b3271d72-1127-4a45-bff1-23b7c80b9c86\") "
Nov 03 13:21:13.062101 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.062075    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-openshift-ca" pod "b3271d72-1127-4a45-bff1-23b7c80b9c86" (UID: "b3271d72-1127-4a45-bff1-23b7c80b9c86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:13.063733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.063709    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-xks-ca" pod "b3271d72-1127-4a45-bff1-23b7c80b9c86" (UID: "b3271d72-1127-4a45-bff1-23b7c80b9c86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:13.063852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.063755    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3271d72-1127-4a45-bff1-23b7c80b9c86-kube-api-access-zpd4c" pod "b3271d72-1127-4a45-bff1-23b7c80b9c86" (UID: "b3271d72-1127-4a45-bff1-23b7c80b9c86"). InnerVolumeSpecName "kube-api-access-zpd4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:13.163050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.163000    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpd4c\" (UniqueName: \"kubernetes.io/projected/b3271d72-1127-4a45-bff1-23b7c80b9c86-kube-api-access-zpd4c\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:13.163050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.163037    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-xks-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:13.163050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.163052    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3271d72-1127-4a45-bff1-23b7c80b9c86-maas-api-openshift-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:13.808395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.808368    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"
Nov 03 13:21:13.808544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.808365    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8" event={"ID":"b3271d72-1127-4a45-bff1-23b7c80b9c86","Type":"ContainerDied","Data":"1a9fcd7671ba314b26d2f9e3ceef544bc20f16d9f2ecd2835e52000864216578"}
Nov 03 13:21:13.808544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:13.808487    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a9fcd7671ba314b26d2f9e3ceef544bc20f16d9f2ecd2835e52000864216578"
Nov 03 13:21:16.780825 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.780740    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"]
Nov 03 13:21:16.781303 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.781289    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b3271d72-1127-4a45-bff1-23b7c80b9c86" containerName="revoke-keys"
Nov 03 13:21:16.781363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.781304    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3271d72-1127-4a45-bff1-23b7c80b9c86" containerName="revoke-keys"
Nov 03 13:21:16.781428 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.781418    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3271d72-1127-4a45-bff1-23b7c80b9c86" containerName="revoke-keys"
Nov 03 13:21:16.785010 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.784984    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.818075 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.818042    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"]
Nov 03 13:21:16.821921 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.821902    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:16.850834 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.850795    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"]
Nov 03 13:21:16.854573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.854541    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 13:21:16.854749 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.854619    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 13:21:16.854749 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.854692    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8strr\""
Nov 03 13:21:16.870803 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.870775    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 13:21:16.905547 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.905512    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.905703 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.905559    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29mv\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-kube-api-access-m29mv\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.905703 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.905691    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.905776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.905737    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.905776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.905762    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.905841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.905791    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.905841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.905806    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.905906 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.905846    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.905906 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.905870    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6c06c3-3ebc-413f-b9f2-3295277b123c-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:16.961790 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.961751    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"]
Nov 03 13:21:16.968925 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.968892    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"]
Nov 03 13:21:16.973729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:16.973704    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.006984 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.006922    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.007165 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.006988    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.007165 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007034    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8f96e5-8722-4503-8361-f5ade2d145f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.007165 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007075    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.007165 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007131    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007384 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007294    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29mv\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-kube-api-access-m29mv\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007410    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007507 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007487    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007564 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007531    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc8f96e5-8722-4503-8361-f5ade2d145f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.007619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007559    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007600    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"]
Nov 03 13:21:17.007713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007602    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007676    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007709    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007738    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zxf\" (UniqueName: \"kubernetes.io/projected/fc8f96e5-8722-4503-8361-f5ade2d145f3-kube-api-access-f9zxf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.007854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007772    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007800    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6c06c3-3ebc-413f-b9f2-3295277b123c-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.007854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.007833    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.008237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.008213    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.008380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.008230    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.008639 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.008614    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.009194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.009171    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6c06c3-3ebc-413f-b9f2-3295277b123c-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.010388 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.010330    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.010578 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.010557    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.038269 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.038193    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.038924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.038902    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29mv\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-kube-api-access-m29mv\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.099042 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.099013    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:17.108931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.108903    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.109109 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.108944    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.109109 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.108980    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.109109 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109010    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.109109 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109055    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8f96e5-8722-4503-8361-f5ade2d145f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.109485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109140    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.109485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109225    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.109485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109291    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6394bce3-b0a3-4f68-af08-0f0fff106087-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.109485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109333    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.109485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109380    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.109485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109408    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.109949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109601    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtzhl\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-kube-api-access-vtzhl\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.109949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109648    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc8f96e5-8722-4503-8361-f5ade2d145f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.109949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109693    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zxf\" (UniqueName: \"kubernetes.io/projected/fc8f96e5-8722-4503-8361-f5ade2d145f3-kube-api-access-f9zxf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.109949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.109738    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.110167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.110097    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.110408 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.110387    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.112067 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.112041    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.112162 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.112131    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc8f96e5-8722-4503-8361-f5ade2d145f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.112162 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.112133    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8f96e5-8722-4503-8361-f5ade2d145f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.113280 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.113255    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.126195 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.125947    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8f96e5-8722-4503-8361-f5ade2d145f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.130928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.130899    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zxf\" (UniqueName: \"kubernetes.io/projected/fc8f96e5-8722-4503-8361-f5ade2d145f3-kube-api-access-f9zxf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hqpn5\" (UID: \"fc8f96e5-8722-4503-8361-f5ade2d145f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.134572 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.134545    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:17.211818 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.211782    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.211915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.211839    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.211915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.211883    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.211992 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.211973    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6394bce3-b0a3-4f68-af08-0f0fff106087-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.212026 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.212017    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.212069 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.212054    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.212117 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.212084    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.212167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.212151    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtzhl\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-kube-api-access-vtzhl\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.212256 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.212240    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.212754 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.212730    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.212802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.212755    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.212802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.212778    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6394bce3-b0a3-4f68-af08-0f0fff106087-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.213049 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.213015    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.213308 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.213282    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.215072 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.215026    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.215240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.215220    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.222555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.222295    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.229198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.228247    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtzhl\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-kube-api-access-vtzhl\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.287939 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.287901    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:17.393926 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.393900    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"]
Nov 03 13:21:17.395994 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:21:17.395963    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6c06c3_3ebc_413f_b9f2_3295277b123c.slice/crio-2296e475a5f782808aa4ae697d692984713eb774a1917f699f3a6c5fb165204d WatchSource:0}: Error finding container 2296e475a5f782808aa4ae697d692984713eb774a1917f699f3a6c5fb165204d: Status 404 returned error can't find the container with id 2296e475a5f782808aa4ae697d692984713eb774a1917f699f3a6c5fb165204d
Nov 03 13:21:17.398151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.398122    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:17.398226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.398201    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:17.398281 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.398227    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:17.434150 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.434125    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"]
Nov 03 13:21:17.435211 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:21:17.435179    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8f96e5_8722_4503_8361_f5ade2d145f3.slice/crio-761f4e5994004f132a0f57e090189e8ad9e3255cca33363d83bfe15b37ef1da5 WatchSource:0}: Error finding container 761f4e5994004f132a0f57e090189e8ad9e3255cca33363d83bfe15b37ef1da5: Status 404 returned error can't find the container with id 761f4e5994004f132a0f57e090189e8ad9e3255cca33363d83bfe15b37ef1da5
Nov 03 13:21:17.576082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.576005    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"]
Nov 03 13:21:17.581979 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:21:17.581940    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6394bce3_b0a3_4f68_af08_0f0fff106087.slice/crio-e910a31fb28406823cfacb762d7e16207cf83e9353932fa43b8405ccd7150df7 WatchSource:0}: Error finding container e910a31fb28406823cfacb762d7e16207cf83e9353932fa43b8405ccd7150df7: Status 404 returned error can't find the container with id e910a31fb28406823cfacb762d7e16207cf83e9353932fa43b8405ccd7150df7
Nov 03 13:21:17.584850 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.584820    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:17.584924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.584889    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:17.584924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.584918    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:17.835140 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.835036    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" event={"ID":"db6c06c3-3ebc-413f-b9f2-3295277b123c","Type":"ContainerStarted","Data":"e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a"}
Nov 03 13:21:17.835140 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.835104    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" event={"ID":"db6c06c3-3ebc-413f-b9f2-3295277b123c","Type":"ContainerStarted","Data":"2296e475a5f782808aa4ae697d692984713eb774a1917f699f3a6c5fb165204d"}
Nov 03 13:21:17.836929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.836899    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5" event={"ID":"fc8f96e5-8722-4503-8361-f5ade2d145f3","Type":"ContainerStarted","Data":"761f4e5994004f132a0f57e090189e8ad9e3255cca33363d83bfe15b37ef1da5"}
Nov 03 13:21:17.838712 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.838684    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" event={"ID":"6394bce3-b0a3-4f68-af08-0f0fff106087","Type":"ContainerStarted","Data":"e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7"}
Nov 03 13:21:17.838825 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.838734    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" event={"ID":"6394bce3-b0a3-4f68-af08-0f0fff106087","Type":"ContainerStarted","Data":"e910a31fb28406823cfacb762d7e16207cf83e9353932fa43b8405ccd7150df7"}
Nov 03 13:21:17.859204 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:17.859152    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podStartSLOduration=1.859136656 podStartE2EDuration="1.859136656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:17.854542508 +0000 UTC m=+765.666210721" watchObservedRunningTime="2026-11-03 13:21:17.859136656 +0000 UTC m=+765.670804844"
Nov 03 13:21:18.100841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:18.100701    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:18.107036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:18.106995    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 13:21:18.107207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:18.107056    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 13:21:18.289501 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:18.289465    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:18.290544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:18.290516    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:21:18.290661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:18.290586    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 13:21:19.101605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:19.101572    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 13:21:19.102103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:19.101619    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 13:21:19.289193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:19.289148    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:21:19.289383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:19.289217    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 13:21:20.101313 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:20.100475    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 13:21:20.101313 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:20.100538    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 13:21:20.288695 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:20.288655    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:21:20.289205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:20.288714    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 13:21:21.100856 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:21.100381    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 13:21:21.100856 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:21.100458    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 13:21:21.288739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:21.288705    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:21:21.289263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:21.288753    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 13:21:22.119385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:22.101412    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 13:21:22.119385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:22.101471    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 13:21:22.289779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:22.289742    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:21:22.290327 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:22.290295    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 13:21:23.111014 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:23.110975    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 13:21:23.114457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:23.114417    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 13:21:23.304368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:23.295775    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:21:23.304368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:23.295836    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 13:21:24.101175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:24.100682    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 13:21:24.101175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:24.100747    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 13:21:24.289172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:24.288713    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:21:24.289172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:24.288775    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 13:21:24.335228 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:24.334942    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk" podUID="4f0e24b4-13ae-451c-9187-78e568c6e1c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:25.021547 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:25.021433    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" podUID="8a27ae67-e693-4daa-a521-ca104653b406" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:25.075370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:25.072442    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk" podUID="4f0e24b4-13ae-451c-9187-78e568c6e1c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:25.105650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:25.105612    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 13:21:25.105917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:25.105883    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 13:21:25.289320 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:25.288804    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:21:25.289320 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:25.288871    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 13:21:25.792909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:25.792872    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:25.793303 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:25.792948    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:26.089810 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.089714    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5" event={"ID":"fc8f96e5-8722-4503-8361-f5ade2d145f3","Type":"ContainerStarted","Data":"6d9e33a71465a44ca8c39e602efeabeea020b9928bb635131659b0a1c8b56639"}
Nov 03 13:21:26.090058 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.089996    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:26.092390 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.092265    4447 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hqpn5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:21:26.092614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.092372    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5" podUID="fc8f96e5-8722-4503-8361-f5ade2d145f3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:21:26.112950 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.112889    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5" podStartSLOduration=1.758268586 podStartE2EDuration="10.112872225s" totalImagesPullingTime="8.354603639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:26.110269917 +0000 UTC m=+773.921938104" watchObservedRunningTime="2026-11-03 13:21:26.112872225 +0000 UTC m=+773.924540412"
Nov 03 13:21:26.114053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.113944    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podStartSLOduration=10.113930944 podStartE2EDuration="10.113930944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:17.880398904 +0000 UTC m=+765.692067106" watchObservedRunningTime="2026-11-03 13:21:26.113930944 +0000 UTC m=+773.925599132"
Nov 03 13:21:26.924818 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.924781    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:26.925311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.925178    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:26.926324 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.926300    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:21:26.985770 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.985699    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:26.987553 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.987528    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:26.990002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:26.989981    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:21:27.096749 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:27.096720    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hqpn5"
Nov 03 13:21:33.411290 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:21:33.411250    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:21:38.007403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:38.007334    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" podUID="8a27ae67-e693-4daa-a521-ca104653b406" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:38.113402 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:38.113336    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg" podUID="875a3afa-0140-40a0-956c-32821edd1f8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 13:21:52.053369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:52.052539    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk" podUID="4f0e24b4-13ae-451c-9187-78e568c6e1c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:54.231901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.231860    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"]
Nov 03 13:21:54.240236 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.240205    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.261253 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.261211    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"]
Nov 03 13:21:54.326977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.326945    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7c192b-openshift-default-dockercfg-rx6fs\""
Nov 03 13:21:54.403577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.403534    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.403577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.403580    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.403799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.403598    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.403799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.403673    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr29g\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-kube-api-access-cr29g\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.403799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.403743    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.403799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.403764    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.403926 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.403855    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.403962 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.403940    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.404025 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.404009    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.505191 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505078    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.505409 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505231    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.505409 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505315    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.505409 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505386    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.505600 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505417    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.505600 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505443    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr29g\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-kube-api-access-cr29g\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.505995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505524    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.506115 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.506027    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.506115 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505793    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.506232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505980    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.506232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.505912    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.506232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.506105    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.506693 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.506674    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.506885 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.506861    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.507879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.507858    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.508229 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.508213    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.512982 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.512951    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.513082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.513045    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr29g\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-kube-api-access-cr29g\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.612416 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.612367    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:54.899990 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:21:54.899959    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7290ec_02e1_41f4_8c59_18aa7f1130dc.slice/crio-4463ee5db4e523653d2b91e574cc75e9d71cb72edeef9f0ee3c4fd5e81c52427 WatchSource:0}: Error finding container 4463ee5db4e523653d2b91e574cc75e9d71cb72edeef9f0ee3c4fd5e81c52427: Status 404 returned error can't find the container with id 4463ee5db4e523653d2b91e574cc75e9d71cb72edeef9f0ee3c4fd5e81c52427
Nov 03 13:21:54.900466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.900434    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"]
Nov 03 13:21:54.902493 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.902466    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:54.902568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.902524    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:54.902568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:54.902552    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:21:55.463934 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:55.463896    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" event={"ID":"8e7290ec-02e1-41f4-8c59-18aa7f1130dc","Type":"ContainerStarted","Data":"720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696"}
Nov 03 13:21:55.464497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:55.463955    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" event={"ID":"8e7290ec-02e1-41f4-8c59-18aa7f1130dc","Type":"ContainerStarted","Data":"4463ee5db4e523653d2b91e574cc75e9d71cb72edeef9f0ee3c4fd5e81c52427"}
Nov 03 13:21:55.480284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:55.480235    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" podStartSLOduration=1.480222033 podStartE2EDuration="1.480222033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:55.480083184 +0000 UTC m=+803.291751393" watchObservedRunningTime="2026-11-03 13:21:55.480222033 +0000 UTC m=+803.291890221"
Nov 03 13:21:55.613371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:55.613243    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:21:55.615014 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:55.614984    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:21:55.615158 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:55.615041    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 13:21:56.613452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:56.613420    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:21:56.613848 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:56.613481    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 13:21:57.613381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:57.613327    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:21:57.613564 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:57.613408    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 13:21:58.613503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:58.613467    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:21:58.613689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:58.613532    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 13:21:59.613430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:59.613389    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:21:59.613610 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:21:59.613446    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 13:22:00.613175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:00.613138    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:22:00.613630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:00.613207    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 13:22:01.617659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:01.617624    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:22:01.618146 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:01.617958    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:22:01.618948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:01.618926    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:22:03.418907 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:22:03.418869    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:22:31.489322 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:31.489140    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"]
Nov 03 13:22:33.417933 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:22:33.417897    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:22:39.315893 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.315859    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"]
Nov 03 13:22:39.320581 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.320555    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.350721 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.350698    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"]
Nov 03 13:22:39.402878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.402811    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.403066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.402916    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.403066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.402997    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.403201 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.403091    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.403201 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.403130    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.403321 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.403246    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.403321 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.403288    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5hf\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-kube-api-access-ht5hf\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.403464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.403444    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.403534 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.403482    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.504974 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.504930    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505016    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505087    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505151    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505186    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505272    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505312    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5hf\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-kube-api-access-ht5hf\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505507 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505412    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505507 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505432    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505507 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505448    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.505928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.505897    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.506104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.506075    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.506271 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.506173    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.506509 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.506481    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.508608 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.508218    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.509221 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.509193    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.513079 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.513027    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.514325 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.514301    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5hf\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-kube-api-access-ht5hf\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:39.670961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:39.670605    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:40.418456 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:40.371981    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:22:40.418456 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:40.372474    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:22:40.418456 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:40.373565    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:22:40.442197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:40.442166    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"]
Nov 03 13:22:40.876727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:40.876633    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" event={"ID":"d3ef6bc9-f218-46fa-9f9a-34a623181d30","Type":"ContainerStarted","Data":"be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545"}
Nov 03 13:22:40.876727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:40.876694    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" event={"ID":"d3ef6bc9-f218-46fa-9f9a-34a623181d30","Type":"ContainerStarted","Data":"ad233b0f69b6d18b3dcf43110a2863d5e95796328c17162371768c75c382eeef"}
Nov 03 13:22:40.894338 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:40.894273    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" podStartSLOduration=1.894256143 podStartE2EDuration="1.894256143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:40.893647671 +0000 UTC m=+848.705315875" watchObservedRunningTime="2026-11-03 13:22:40.894256143 +0000 UTC m=+848.705924333"
Nov 03 13:22:41.674110 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:41.674072    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:41.675763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:41.675738    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 13:22:41.675881 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:41.675784    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 13:22:42.673465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:42.673433    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 13:22:42.673658 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:42.673496    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 13:22:43.673746 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:43.673711    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 13:22:43.674139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:43.673768    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 13:22:44.673937 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:44.673896    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 13:22:44.674542 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:44.673965    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 13:22:45.673783 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:45.673748    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 13:22:45.674003 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:45.673809    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 13:22:46.525072 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.524966    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk" podUID="4f0e24b4-13ae-451c-9187-78e568c6e1c8" containerName="main" containerID="cri-o://865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751" gracePeriod=45
Nov 03 13:22:46.674454 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.674422    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 13:22:46.674793 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.674478    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 13:22:46.858666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.858629    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:22:46.957962 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.957928    4447 generic.go:356] "Generic (PLEG): container finished" podID="4f0e24b4-13ae-451c-9187-78e568c6e1c8" containerID="865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751" exitCode=0
Nov 03 13:22:46.958137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.957969    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk" event={"ID":"4f0e24b4-13ae-451c-9187-78e568c6e1c8","Type":"ContainerDied","Data":"865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751"}
Nov 03 13:22:46.958137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.958001    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk" event={"ID":"4f0e24b4-13ae-451c-9187-78e568c6e1c8","Type":"ContainerDied","Data":"5d666ba01037a946e0cbcf6e06fed396e91ca6957c38f16ec43cb0097d952503"}
Nov 03 13:22:46.958137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.958018    4447 scope.go:122] "RemoveContainer" containerID="865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751"
Nov 03 13:22:46.958137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.957972    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"
Nov 03 13:22:46.965268 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.965238    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0e24b4-13ae-451c-9187-78e568c6e1c8-kube-api-access-s6s4l\" (UniqueName: \"kubernetes.io/projected/4f0e24b4-13ae-451c-9187-78e568c6e1c8-kube-api-access-s6s4l\") pod \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") "
Nov 03 13:22:46.965444 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.965296    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-home\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-home\") pod \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") "
Nov 03 13:22:46.965444 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.965415    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-dshm\") pod \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") "
Nov 03 13:22:46.965577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.965452    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-model-cache\") pod \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") "
Nov 03 13:22:46.965577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.965480    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tls-certs\") pod \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") "
Nov 03 13:22:46.965577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.965560    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-home" pod "4f0e24b4-13ae-451c-9187-78e568c6e1c8" (UID: "4f0e24b4-13ae-451c-9187-78e568c6e1c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:22:46.965735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.965593    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tmp-dir\") pod \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\" (UID: \"4f0e24b4-13ae-451c-9187-78e568c6e1c8\") "
Nov 03 13:22:46.965735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.965667    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-model-cache" pod "4f0e24b4-13ae-451c-9187-78e568c6e1c8" (UID: "4f0e24b4-13ae-451c-9187-78e568c6e1c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:22:46.965956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.965934    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tmp-dir" pod "4f0e24b4-13ae-451c-9187-78e568c6e1c8" (UID: "4f0e24b4-13ae-451c-9187-78e568c6e1c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:22:46.966194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.966181    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-home\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:46.966277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.966202    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-model-cache\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:46.966277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.966215    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tmp-dir\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:46.968121 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.968098    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-dshm" pod "4f0e24b4-13ae-451c-9187-78e568c6e1c8" (UID: "4f0e24b4-13ae-451c-9187-78e568c6e1c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:22:46.968248 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.968226    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tls-certs" pod "4f0e24b4-13ae-451c-9187-78e568c6e1c8" (UID: "4f0e24b4-13ae-451c-9187-78e568c6e1c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:46.968314 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.968276    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0e24b4-13ae-451c-9187-78e568c6e1c8-kube-api-access-s6s4l" pod "4f0e24b4-13ae-451c-9187-78e568c6e1c8" (UID: "4f0e24b4-13ae-451c-9187-78e568c6e1c8"). InnerVolumeSpecName "kube-api-access-s6s4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:46.971189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.971169    4447 scope.go:122] "RemoveContainer" containerID="865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751"
Nov 03 13:22:46.971542 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:22:46.971512    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751\": container with ID starting with 865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751 not found: ID does not exist" containerID="865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751"
Nov 03 13:22:46.971652 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:46.971553    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751"} err="failed to get container status \"865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751\": rpc error: code = NotFound desc = could not find container \"865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751\": container with ID starting with 865aec62c02145af661caf4abb306954567b1842353a99882de969beac504751 not found: ID does not exist"
Nov 03 13:22:47.068089 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:47.067989    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6s4l\" (UniqueName: \"kubernetes.io/projected/4f0e24b4-13ae-451c-9187-78e568c6e1c8-kube-api-access-s6s4l\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:47.068089 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:47.068029    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0e24b4-13ae-451c-9187-78e568c6e1c8-dshm\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:47.068089 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:47.068044    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0e24b4-13ae-451c-9187-78e568c6e1c8-tls-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:47.285717 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:47.285680    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"]
Nov 03 13:22:47.291355 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:47.291314    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-17adbf/test-model-17adbf-kserve-b549bbfd9-6kqhk"]
Nov 03 13:22:47.678798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:47.678767    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:47.679219 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:47.679130    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:47.679928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:47.679909    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:22:48.319651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:22:48.319617    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0e24b4-13ae-451c-9187-78e568c6e1c8" path="/var/lib/kubelet/pods/4f0e24b4-13ae-451c-9187-78e568c6e1c8/volumes"
Nov 03 13:23:03.412281 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:23:03.412236    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:23:13.800740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:13.800441    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"]
Nov 03 13:23:13.800740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:13.800705    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27" podUID="c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" containerName="maas-api" containerID="cri-o://191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c" gracePeriod=30
Nov 03 13:23:13.887110 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:13.887070    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27" podUID="c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.56:8443/health\": dial tcp 10.133.0.56:8443: connect: connection refused"
Nov 03 13:23:14.157440 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.157400    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:23:14.212610 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.212573    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-metrics-certs\") pod \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") "
Nov 03 13:23:14.212812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.212703    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-kube-api-access-8kn29\" (UniqueName: \"kubernetes.io/projected/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-kube-api-access-8kn29\") pod \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") "
Nov 03 13:23:14.212812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.212742    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-openshift-service-ca\") pod \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") "
Nov 03 13:23:14.212937 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.212869    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls\") pod \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\" (UID: \"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c\") "
Nov 03 13:23:14.213135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.213111    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-openshift-service-ca" pod "c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" (UID: "c1c457a9-8beb-4fca-aaaa-241f3cc4e49c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:14.213489 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.213465    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:14.214903 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.214877    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls" pod "c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" (UID: "c1c457a9-8beb-4fca-aaaa-241f3cc4e49c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:14.215001 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.214936    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-kube-api-access-8kn29" pod "c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" (UID: "c1c457a9-8beb-4fca-aaaa-241f3cc4e49c"). InnerVolumeSpecName "kube-api-access-8kn29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:14.215045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.215005    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-metrics-certs" pod "c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" (UID: "c1c457a9-8beb-4fca-aaaa-241f3cc4e49c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:14.271691 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.271658    4447 generic.go:356] "Generic (PLEG): container finished" podID="c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" containerID="191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c" exitCode=0
Nov 03 13:23:14.271889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.271704    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"
Nov 03 13:23:14.271889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.271737    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27" event={"ID":"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c","Type":"ContainerDied","Data":"191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c"}
Nov 03 13:23:14.271889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.271800    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27" event={"ID":"c1c457a9-8beb-4fca-aaaa-241f3cc4e49c","Type":"ContainerDied","Data":"4f1694dea1b81a624f5e23ab5107ee3adf38ff9c51b82a02cbbc2835e7838ace"}
Nov 03 13:23:14.271889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.271830    4447 scope.go:122] "RemoveContainer" containerID="191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c"
Nov 03 13:23:14.284878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.284835    4447 scope.go:122] "RemoveContainer" containerID="191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c"
Nov 03 13:23:14.285281 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:23:14.285234    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c\": container with ID starting with 191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c not found: ID does not exist" containerID="191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c"
Nov 03 13:23:14.285438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.285283    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c"} err="failed to get container status \"191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c\": rpc error: code = NotFound desc = could not find container \"191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c\": container with ID starting with 191a3c709860ffaad313cbf574bedeabb1413048fdd4d7c157d5247e2bc6b60c not found: ID does not exist"
Nov 03 13:23:14.303414 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.301750    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"]
Nov 03 13:23:14.306970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.306943    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db71f4-86d869785c-4wn27"]
Nov 03 13:23:14.314555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.314502    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:14.314555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.314534    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kn29\" (UniqueName: \"kubernetes.io/projected/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-kube-api-access-8kn29\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:14.314555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.314550    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:14.321994 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:14.321966    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" path="/var/lib/kubelet/pods/c1c457a9-8beb-4fca-aaaa-241f3cc4e49c/volumes"
Nov 03 13:23:29.985367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:29.984426    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" podUID="8a27ae67-e693-4daa-a521-ca104653b406" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:30.014367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:30.013448    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" podUID="8a27ae67-e693-4daa-a521-ca104653b406" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:31.102369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.100459    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"]
Nov 03 13:23:31.102369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.101447    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" containerName="maas-api"
Nov 03 13:23:31.102369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.101465    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" containerName="maas-api"
Nov 03 13:23:31.102369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.101478    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4f0e24b4-13ae-451c-9187-78e568c6e1c8" containerName="main"
Nov 03 13:23:31.102369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.101486    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0e24b4-13ae-451c-9187-78e568c6e1c8" containerName="main"
Nov 03 13:23:31.102369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.101622    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c457a9-8beb-4fca-aaaa-241f3cc4e49c" containerName="maas-api"
Nov 03 13:23:31.102369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.101636    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0e24b4-13ae-451c-9187-78e568c6e1c8" containerName="main"
Nov 03 13:23:31.116011 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.105790    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.116444 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.116419    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"]
Nov 03 13:23:31.119805 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.119784    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d3ca48-serving-cert\""
Nov 03 13:23:31.226702 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.226671    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-metrics-certs\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.226857 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.226715    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e665799-44f7-4a3d-9d6f-d9166accd2c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.226937 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.226912    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.226988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.226970    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rb7\" (UniqueName: \"kubernetes.io/projected/4e665799-44f7-4a3d-9d6f-d9166accd2c9-kube-api-access-x8rb7\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.328721 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.328684    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.328932 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.328738    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rb7\" (UniqueName: \"kubernetes.io/projected/4e665799-44f7-4a3d-9d6f-d9166accd2c9-kube-api-access-x8rb7\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.329146 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.328937    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-metrics-certs\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.329146 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.328977    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e665799-44f7-4a3d-9d6f-d9166accd2c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.330328 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.330301    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e665799-44f7-4a3d-9d6f-d9166accd2c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.332863 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.332835    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-metrics-certs\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.332971 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.332898    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.337216 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.337188    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rb7\" (UniqueName: \"kubernetes.io/projected/4e665799-44f7-4a3d-9d6f-d9166accd2c9-kube-api-access-x8rb7\") pod \"maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.435232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.434495    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:31.734609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:31.734560    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"]
Nov 03 13:23:31.736899 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:23:31.736857    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e665799_44f7_4a3d_9d6f_d9166accd2c9.slice/crio-3bceee7070416057acdc9017e7b7d6f0365434cb7dccaa8144321325fdbaa9a6 WatchSource:0}: Error finding container 3bceee7070416057acdc9017e7b7d6f0365434cb7dccaa8144321325fdbaa9a6: Status 404 returned error can't find the container with id 3bceee7070416057acdc9017e7b7d6f0365434cb7dccaa8144321325fdbaa9a6
Nov 03 13:23:32.453073 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:32.452981    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt" event={"ID":"4e665799-44f7-4a3d-9d6f-d9166accd2c9","Type":"ContainerStarted","Data":"c609794f0083036ec9928259fd4641f1b6fe4b325af4ecf996fbd8eb41300a16"}
Nov 03 13:23:32.453073 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:32.453038    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt" event={"ID":"4e665799-44f7-4a3d-9d6f-d9166accd2c9","Type":"ContainerStarted","Data":"3bceee7070416057acdc9017e7b7d6f0365434cb7dccaa8144321325fdbaa9a6"}
Nov 03 13:23:32.453513 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:32.453184    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:23:32.478462 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:32.478403    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt" podStartSLOduration=1.016976683 podStartE2EDuration="1.478384559s" totalImagesPullingTime="461.407876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:32.474986373 +0000 UTC m=+900.286654567" watchObservedRunningTime="2026-11-03 13:23:32.478384559 +0000 UTC m=+900.290052748"
Nov 03 13:23:33.410951 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:23:33.410906    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:23:38.465312 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:23:38.465280    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:24:03.412861 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:24:03.412820    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:24:24.532553 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.532454    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"]
Nov 03 13:24:24.547587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.542938    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.548674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.548631    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"]
Nov 03 13:24:24.662870 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.662834    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.663055 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.662893    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.663055 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.662976    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.663055 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.663002    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.663168 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.663102    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.663168 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.663130    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.663168 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.663148    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.663289 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.663227    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.663289 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.663264    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4kzr\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-kube-api-access-k4kzr\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764126    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764390 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764174    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764390 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764277    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764390 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764332    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764565 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764397    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4kzr\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-kube-api-access-k4kzr\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764565 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764493    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764558    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764627    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764635    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764714    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764767    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.764945 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.764853    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.765095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.765073    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.765447 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.765421    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.766773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.766749    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.767097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.767078    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.780530 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.780502    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.780648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.780630    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4kzr\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-kube-api-access-k4kzr\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:24.879836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:24.879743    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:25.173501 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:25.173470    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"]
Nov 03 13:24:25.174618 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:24:25.174586    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484ecf48_e4c0_4175_b9b3_7a03e31bc001.slice/crio-e8701dbbb1f6cdb53b2f9731e94cdc2ffbe4e5f2a82ab77b5cc692e6f3f0fd5b WatchSource:0}: Error finding container e8701dbbb1f6cdb53b2f9731e94cdc2ffbe4e5f2a82ab77b5cc692e6f3f0fd5b: Status 404 returned error can't find the container with id e8701dbbb1f6cdb53b2f9731e94cdc2ffbe4e5f2a82ab77b5cc692e6f3f0fd5b
Nov 03 13:24:25.176898 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:25.176864    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:24:25.176997 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:25.176942    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:24:25.177070 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:25.177015    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:24:25.992868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:25.992816    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" event={"ID":"484ecf48-e4c0-4175-b9b3-7a03e31bc001","Type":"ContainerStarted","Data":"ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9"}
Nov 03 13:24:25.993435 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:25.992897    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" event={"ID":"484ecf48-e4c0-4175-b9b3-7a03e31bc001","Type":"ContainerStarted","Data":"e8701dbbb1f6cdb53b2f9731e94cdc2ffbe4e5f2a82ab77b5cc692e6f3f0fd5b"}
Nov 03 13:24:26.020731 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:26.020676    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" podStartSLOduration=2.020659692 podStartE2EDuration="2.020659692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:26.016389389 +0000 UTC m=+953.828057588" watchObservedRunningTime="2026-11-03 13:24:26.020659692 +0000 UTC m=+953.832327879"
Nov 03 13:24:26.889421 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:26.889323    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:26.889625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:26.889585    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 13:24:26.889687 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:26.889640    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 13:24:27.880814 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:27.880776    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 13:24:27.881839 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:27.881795    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 13:24:28.881471 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:28.881436    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 13:24:28.881978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:28.881499    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 13:24:29.880642 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:29.880604    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 13:24:29.880821 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:29.880681    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 13:24:30.880618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:30.880572    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 13:24:30.881087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:30.880626    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 13:24:32.348648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:32.348615    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:32.349429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:32.349409    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:32.350226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:24:32.350207    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:24:33.413329 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:24:33.413286    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:25:03.411859 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:25:03.411823    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:25:12.594936 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.594899    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"]
Nov 03 13:25:12.602082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.602057    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.608876 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.608850    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-021a5a5c-serving-cert\""
Nov 03 13:25:12.615783 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.615758    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"]
Nov 03 13:25:12.756128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.756090    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-maas-api-tls\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.756128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.756128    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-metrics-certs\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.756410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.756149    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-openshift-service-ca\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.756575 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.756552    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22w7h\" (UniqueName: \"kubernetes.io/projected/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-kube-api-access-22w7h\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.857910 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.857817    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22w7h\" (UniqueName: \"kubernetes.io/projected/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-kube-api-access-22w7h\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.857910 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.857871    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-maas-api-tls\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.858142 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.858009    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-metrics-certs\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.858142 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.858044    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-openshift-service-ca\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.858943 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.858917    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-openshift-service-ca\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.860558 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.860533    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-maas-api-tls\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.860685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.860558    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-metrics-certs\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.867403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.867380    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22w7h\" (UniqueName: \"kubernetes.io/projected/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-kube-api-access-22w7h\") pod \"maas-api-e2e-ait-021a5a5c-9cc545477-99855\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:12.924891 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:12.924859    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:13.234702 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:13.233441    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"]
Nov 03 13:25:13.243039 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:25:13.242993    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33a1a95_da3e_4ada_8de3_6d20b0eedd5b.slice/crio-c141093f531c6c96edeb2ecf6076433ceff2d2ec20c404990bcee7f74ea1c3dd WatchSource:0}: Error finding container c141093f531c6c96edeb2ecf6076433ceff2d2ec20c404990bcee7f74ea1c3dd: Status 404 returned error can't find the container with id c141093f531c6c96edeb2ecf6076433ceff2d2ec20c404990bcee7f74ea1c3dd
Nov 03 13:25:13.245405 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:13.245325    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:25:13.476361 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:13.476321    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855" event={"ID":"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b","Type":"ContainerStarted","Data":"c141093f531c6c96edeb2ecf6076433ceff2d2ec20c404990bcee7f74ea1c3dd"}
Nov 03 13:25:14.486721 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:14.486685    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855" event={"ID":"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b","Type":"ContainerStarted","Data":"2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae"}
Nov 03 13:25:14.487144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:14.486862    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:14.508931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:14.508871    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855" podStartSLOduration=2.129416433 podStartE2EDuration="2.508855706s" totalImagesPullingTime="379.439273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:14.50747215 +0000 UTC m=+1002.319140341" watchObservedRunningTime="2026-11-03 13:25:14.508855706 +0000 UTC m=+1002.320523894"
Nov 03 13:25:20.504999 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:20.504971    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:30.002151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.002112    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"]
Nov 03 13:25:30.007275 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.007247    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.018105 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.016405    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-8f8aa8-serving-cert\""
Nov 03 13:25:30.027998 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.027968    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"]
Nov 03 13:25:30.167730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.167687    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.167909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.167809    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.167909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.167860    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.167909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.167884    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9mn\" (UniqueName: \"kubernetes.io/projected/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-kube-api-access-4g9mn\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.269505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.269420    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.269505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.269470    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.269716 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.269508    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.269716 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.269531    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9mn\" (UniqueName: \"kubernetes.io/projected/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-kube-api-access-4g9mn\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.270358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.270312    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.272041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.272022    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.272189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.272168    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.286997 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.286968    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9mn\" (UniqueName: \"kubernetes.io/projected/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-kube-api-access-4g9mn\") pod \"maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.341611 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.341577    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:30.627685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:30.627658    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"]
Nov 03 13:25:30.628962 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:25:30.628938    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c4cac3_b6f0_4f9f_b6fd_20105519b6c8.slice/crio-a67717bfc8c44e6981b081148a7e3bc561f8ea7c7496c7ad41beb5792bc5d15e WatchSource:0}: Error finding container a67717bfc8c44e6981b081148a7e3bc561f8ea7c7496c7ad41beb5792bc5d15e: Status 404 returned error can't find the container with id a67717bfc8c44e6981b081148a7e3bc561f8ea7c7496c7ad41beb5792bc5d15e
Nov 03 13:25:31.622232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:31.622190    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm" event={"ID":"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8","Type":"ContainerStarted","Data":"a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea"}
Nov 03 13:25:31.622753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:31.622292    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm" event={"ID":"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8","Type":"ContainerStarted","Data":"a67717bfc8c44e6981b081148a7e3bc561f8ea7c7496c7ad41beb5792bc5d15e"}
Nov 03 13:25:31.622753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:31.622551    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:31.640971 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:31.640919    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm" podStartSLOduration=2.172577412 podStartE2EDuration="2.640904023s" totalImagesPullingTime="468.326611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:31.637182365 +0000 UTC m=+1019.448850552" watchObservedRunningTime="2026-11-03 13:25:31.640904023 +0000 UTC m=+1019.452572211"
Nov 03 13:25:33.410487 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:25:33.410449    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:25:37.107260 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.107224    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"]
Nov 03 13:25:37.108053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.108017    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855" podUID="b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" containerName="maas-api" containerID="cri-o://2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae" gracePeriod=30
Nov 03 13:25:37.408339 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.408306    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:37.472686 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.471310    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-kube-api-access-22w7h\" (UniqueName: \"kubernetes.io/projected/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-kube-api-access-22w7h\") pod \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") "
Nov 03 13:25:37.472880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.472767    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-openshift-service-ca\") pod \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") "
Nov 03 13:25:37.472880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.472825    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-metrics-certs\") pod \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") "
Nov 03 13:25:37.473013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.472879    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-maas-api-tls\") pod \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\" (UID: \"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b\") "
Nov 03 13:25:37.473678 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.473159    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-openshift-service-ca" pod "b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" (UID: "b33a1a95-da3e-4ada-8de3-6d20b0eedd5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:37.473678 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.473571    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:37.480336 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.480100    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-metrics-certs" pod "b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" (UID: "b33a1a95-da3e-4ada-8de3-6d20b0eedd5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:25:37.483606 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.483561    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-maas-api-tls" pod "b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" (UID: "b33a1a95-da3e-4ada-8de3-6d20b0eedd5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:25:37.484003 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.483962    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-kube-api-access-22w7h" pod "b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" (UID: "b33a1a95-da3e-4ada-8de3-6d20b0eedd5b"). InnerVolumeSpecName "kube-api-access-22w7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:37.574371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.574283    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:37.574371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.574309    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22w7h\" (UniqueName: \"kubernetes.io/projected/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-kube-api-access-22w7h\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:37.574371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.574320    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:37.637662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.637564    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:25:37.692265 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.692236    4447 generic.go:356] "Generic (PLEG): container finished" podID="b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" containerID="2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae" exitCode=0
Nov 03 13:25:37.693329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.692486    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855" event={"ID":"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b","Type":"ContainerDied","Data":"2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae"}
Nov 03 13:25:37.693329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.692681    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855" event={"ID":"b33a1a95-da3e-4ada-8de3-6d20b0eedd5b","Type":"ContainerDied","Data":"c141093f531c6c96edeb2ecf6076433ceff2d2ec20c404990bcee7f74ea1c3dd"}
Nov 03 13:25:37.693329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.692528    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"
Nov 03 13:25:37.693329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.692716    4447 scope.go:122] "RemoveContainer" containerID="2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae"
Nov 03 13:25:37.713785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.713669    4447 scope.go:122] "RemoveContainer" containerID="2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae"
Nov 03 13:25:37.714043 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:25:37.714019    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae\": container with ID starting with 2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae not found: ID does not exist" containerID="2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae"
Nov 03 13:25:37.714131 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.714057    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae"} err="failed to get container status \"2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae\": rpc error: code = NotFound desc = could not find container \"2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae\": container with ID starting with 2bf1bc1d92661686de72fe1477a266ffdb4347a86089cf9fc352a47bd4483cae not found: ID does not exist"
Nov 03 13:25:37.733300 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.733276    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"]
Nov 03 13:25:37.736424 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:37.736402    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-021a5a5c-9cc545477-99855"]
Nov 03 13:25:38.321601 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:38.321570    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" path="/var/lib/kubelet/pods/b33a1a95-da3e-4ada-8de3-6d20b0eedd5b/volumes"
Nov 03 13:25:44.112380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.112334    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"]
Nov 03 13:25:44.112987 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.112967    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" containerName="maas-api"
Nov 03 13:25:44.112987 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.112986    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" containerName="maas-api"
Nov 03 13:25:44.113177 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.113122    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33a1a95-da3e-4ada-8de3-6d20b0eedd5b" containerName="maas-api"
Nov 03 13:25:44.117899 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.117881    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:44.124941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.124917    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2dbf739c-dockercfg-jg9xr\""
Nov 03 13:25:44.129470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.129442    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"]
Nov 03 13:25:44.132605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.132578    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2dbf739c\""
Nov 03 13:25:44.290368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.286028    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaedc3-94f7-478c-b0ac-aac35df2b25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd\" (UID: \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:44.290368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.286144    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxctm\" (UniqueName: \"kubernetes.io/projected/50eaedc3-94f7-478c-b0ac-aac35df2b25e-kube-api-access-zxctm\") pod \"payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd\" (UID: \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:44.387182 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.387084    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaedc3-94f7-478c-b0ac-aac35df2b25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd\" (UID: \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:44.387182 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.387158    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxctm\" (UniqueName: \"kubernetes.io/projected/50eaedc3-94f7-478c-b0ac-aac35df2b25e-kube-api-access-zxctm\") pod \"payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd\" (UID: \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:44.387863 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.387840    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaedc3-94f7-478c-b0ac-aac35df2b25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd\" (UID: \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:44.395640 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.395615    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxctm\" (UniqueName: \"kubernetes.io/projected/50eaedc3-94f7-478c-b0ac-aac35df2b25e-kube-api-access-zxctm\") pod \"payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd\" (UID: \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:44.438322 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.438291    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:44.728379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.727996    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"]
Nov 03 13:25:44.728379 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:25:44.728251    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eaedc3_94f7_478c_b0ac_aac35df2b25e.slice/crio-d1471c982e06ecc0aa1ca50019af0d70dd2e8aceb1c0a697594e94bee9fefe54 WatchSource:0}: Error finding container d1471c982e06ecc0aa1ca50019af0d70dd2e8aceb1c0a697594e94bee9fefe54: Status 404 returned error can't find the container with id d1471c982e06ecc0aa1ca50019af0d70dd2e8aceb1c0a697594e94bee9fefe54
Nov 03 13:25:44.752492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:44.752457    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd" event={"ID":"50eaedc3-94f7-478c-b0ac-aac35df2b25e","Type":"ContainerStarted","Data":"d1471c982e06ecc0aa1ca50019af0d70dd2e8aceb1c0a697594e94bee9fefe54"}
Nov 03 13:25:46.470174 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.470134    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"]
Nov 03 13:25:46.495335 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.495296    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"]
Nov 03 13:25:46.495553 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.495448    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.499919 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.499891    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"routing-overlay\""
Nov 03 13:25:46.501504 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.500451    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"openshift-service-ca.crt\""
Nov 03 13:25:46.501504 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.500583    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-e2e--cd1da41a-dockercfg-txc88\""
Nov 03 13:25:46.501504 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.500662    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-29dd648f7d07\""
Nov 03 13:25:46.501504 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.500729    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"kube-root-ca.crt\""
Nov 03 13:25:46.501504 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.500781    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c192b\""
Nov 03 13:25:46.501504 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.500851    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-f5ffe280c5d9\""
Nov 03 13:25:46.501504 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.500919    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-384e44d183e3\""
Nov 03 13:25:46.632678 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.632627    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.632871 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.632693    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.632871 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.632793    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.632978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.632966    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gvhj\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-kube-api-access-7gvhj\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.734643 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.734546    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.734643 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.734619    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.734841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.734674    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.734888 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.734844    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gvhj\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-kube-api-access-7gvhj\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.735812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.735781    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.736012 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.735822    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.737866 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.737839    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.743924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.743896    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gvhj\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-kube-api-access-7gvhj\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:46.829988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:46.829946    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:47.158138 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:47.158099    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"]
Nov 03 13:25:47.164410 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:25:47.164339    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6763dd66_263e_4b5b_9f9e_57a269ca1493.slice/crio-f1c21b842736a9ade44c4d3590f86db4a8399ac0c4e045b42eb7108b9168f0ba WatchSource:0}: Error finding container f1c21b842736a9ade44c4d3590f86db4a8399ac0c4e045b42eb7108b9168f0ba: Status 404 returned error can't find the container with id f1c21b842736a9ade44c4d3590f86db4a8399ac0c4e045b42eb7108b9168f0ba
Nov 03 13:25:47.783961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:47.783820    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8" event={"ID":"6763dd66-263e-4b5b-9f9e-57a269ca1493","Type":"ContainerStarted","Data":"db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b"}
Nov 03 13:25:47.783961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:47.783892    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:47.783961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:47.783917    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8" event={"ID":"6763dd66-263e-4b5b-9f9e-57a269ca1493","Type":"ContainerStarted","Data":"f1c21b842736a9ade44c4d3590f86db4a8399ac0c4e045b42eb7108b9168f0ba"}
Nov 03 13:25:47.785607 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:47.785578    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd" event={"ID":"50eaedc3-94f7-478c-b0ac-aac35df2b25e","Type":"ContainerStarted","Data":"7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36"}
Nov 03 13:25:47.785728 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:47.785634    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:47.805061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:47.805008    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8" podStartSLOduration=1.428678219 podStartE2EDuration="1.804993209s" totalImagesPullingTime="376.31499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:47.803584735 +0000 UTC m=+1035.615252939" watchObservedRunningTime="2026-11-03 13:25:47.804993209 +0000 UTC m=+1035.616661594"
Nov 03 13:25:47.835233 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:47.835182    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd" podStartSLOduration=1.086040954 podStartE2EDuration="3.835168067s" totalImagesPullingTime="2.749127113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:47.830737666 +0000 UTC m=+1035.642405854" watchObservedRunningTime="2026-11-03 13:25:47.835168067 +0000 UTC m=+1035.646836255"
Nov 03 13:25:48.501990 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.501953    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"]
Nov 03 13:25:48.506394 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.506372    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:25:48.512761 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.512737    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-9add72\""
Nov 03 13:25:48.512761 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.512734    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-9add72-dockercfg-wdhth\""
Nov 03 13:25:48.517214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.517188    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"]
Nov 03 13:25:48.571498 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.571458    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt4j4\" (UniqueName: \"kubernetes.io/projected/f56fb04c-d9fe-48e9-b361-2a93fb587f71-kube-api-access-dt4j4\") pod \"payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf\" (UID: \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:25:48.571685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.571618    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56fb04c-d9fe-48e9-b361-2a93fb587f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf\" (UID: \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:25:48.672869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.672825    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt4j4\" (UniqueName: \"kubernetes.io/projected/f56fb04c-d9fe-48e9-b361-2a93fb587f71-kube-api-access-dt4j4\") pod \"payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf\" (UID: \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:25:48.673080 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.672966    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56fb04c-d9fe-48e9-b361-2a93fb587f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf\" (UID: \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:25:48.674303 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.673968    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56fb04c-d9fe-48e9-b361-2a93fb587f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf\" (UID: \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:25:48.689209 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.689174    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt4j4\" (UniqueName: \"kubernetes.io/projected/f56fb04c-d9fe-48e9-b361-2a93fb587f71-kube-api-access-dt4j4\") pod \"payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf\" (UID: \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:25:48.823541 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:48.823443    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:25:49.115037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:49.113749    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"]
Nov 03 13:25:49.117113 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:25:49.117082    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56fb04c_d9fe_48e9_b361_2a93fb587f71.slice/crio-997c7af65fa6cd0d63f4aab2bcc0d2a49bbdfc1d04c0874040cc66d3844ce7f7 WatchSource:0}: Error finding container 997c7af65fa6cd0d63f4aab2bcc0d2a49bbdfc1d04c0874040cc66d3844ce7f7: Status 404 returned error can't find the container with id 997c7af65fa6cd0d63f4aab2bcc0d2a49bbdfc1d04c0874040cc66d3844ce7f7
Nov 03 13:25:49.799415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:49.799374    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf" event={"ID":"f56fb04c-d9fe-48e9-b361-2a93fb587f71","Type":"ContainerStarted","Data":"7c4036ac2cb6ef86cd5834dc5c7141198d9423b89e917cc8fb22136b01722852"}
Nov 03 13:25:49.799596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:49.799431    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf" event={"ID":"f56fb04c-d9fe-48e9-b361-2a93fb587f71","Type":"ContainerStarted","Data":"997c7af65fa6cd0d63f4aab2bcc0d2a49bbdfc1d04c0874040cc66d3844ce7f7"}
Nov 03 13:25:49.799596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:49.799583    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:25:49.820011 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:49.819960    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf" podStartSLOduration=1.8199456280000001 podStartE2EDuration="1.819945628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:49.818466822 +0000 UTC m=+1037.630135034" watchObservedRunningTime="2026-11-03 13:25:49.819945628 +0000 UTC m=+1037.631613815"
Nov 03 13:25:51.170179 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.167082    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"]
Nov 03 13:25:51.172322 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.172295    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.179425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.179385    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d3ca48-openshift-default-dockercfg-lgz27\""
Nov 03 13:25:51.186689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.186653    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"]
Nov 03 13:25:51.203039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.202593    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.203039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.202674    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.203039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.202702    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.203039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.202743    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.203039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.202771    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.203039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.202835    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.203039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.202918    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.203883 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.203633    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.203883 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.203683    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkqk\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-kube-api-access-tvkqk\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.320681 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.320638    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.320879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.320696    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.321020 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.320914    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.321198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.321163    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.322915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.321883    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.322915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.322076    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.322915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.322289    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.322915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.322316    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkqk\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-kube-api-access-tvkqk\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.323214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.323153    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.324083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.324057    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.324185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.324167    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.324245 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.324197    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.326255 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.324513    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.326255 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.324514    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.337405 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.333306    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.381408 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.375646    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.398564 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.398532    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.406367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.402544    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkqk\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-kube-api-access-tvkqk\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.508707 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.508616    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:51.860099 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.860011    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"]
Nov 03 13:25:51.863478 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:25:51.863451    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4036ac2c_02b8_4eed_9efa_59b749a95b8b.slice/crio-c08175666ee091089580760aa7c3a41b128b47934749c6c08db873c8bb7e4f61 WatchSource:0}: Error finding container c08175666ee091089580760aa7c3a41b128b47934749c6c08db873c8bb7e4f61: Status 404 returned error can't find the container with id c08175666ee091089580760aa7c3a41b128b47934749c6c08db873c8bb7e4f61
Nov 03 13:25:51.866189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.866156    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:25:51.866276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.866223    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:25:51.866276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:51.866253    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:25:52.350219 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.350187    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"]
Nov 03 13:25:52.360879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.360841    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"]
Nov 03 13:25:52.361081 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.360972    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.368721 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.368689    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-2dbf739c-serving-cert\""
Nov 03 13:25:52.462174 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.462113    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c76fdca-f0ac-418c-80df-623a97134eae-openshift-service-ca\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.464803 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.464772    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-maas-api-tls\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.465217 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.465197    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-metrics-certs\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.465496 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.465477    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrgd\" (UniqueName: \"kubernetes.io/projected/0c76fdca-f0ac-418c-80df-623a97134eae-kube-api-access-lhrgd\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.569141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.569102    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-metrics-certs\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.569451 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.569429    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrgd\" (UniqueName: \"kubernetes.io/projected/0c76fdca-f0ac-418c-80df-623a97134eae-kube-api-access-lhrgd\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.569683 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.569667    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c76fdca-f0ac-418c-80df-623a97134eae-openshift-service-ca\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.569834 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.569820    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-maas-api-tls\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.574887 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.574493    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c76fdca-f0ac-418c-80df-623a97134eae-openshift-service-ca\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.600465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.589565    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-metrics-certs\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.600465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.595656    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrgd\" (UniqueName: \"kubernetes.io/projected/0c76fdca-f0ac-418c-80df-623a97134eae-kube-api-access-lhrgd\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.602028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.601979    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-maas-api-tls\") pod \"maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.689801 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.689759    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:52.822271 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.822239    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" event={"ID":"4036ac2c-02b8-4eed-9efa-59b749a95b8b","Type":"ContainerStarted","Data":"0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d"}
Nov 03 13:25:52.822477 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.822451    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" event={"ID":"4036ac2c-02b8-4eed-9efa-59b749a95b8b","Type":"ContainerStarted","Data":"c08175666ee091089580760aa7c3a41b128b47934749c6c08db873c8bb7e4f61"}
Nov 03 13:25:52.916455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:52.911644    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" podStartSLOduration=1.9116234570000001 podStartE2EDuration="1.911623457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:52.862137767 +0000 UTC m=+1040.673805982" watchObservedRunningTime="2026-11-03 13:25:52.911623457 +0000 UTC m=+1040.723291646"
Nov 03 13:25:53.025290 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:53.025255    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"]
Nov 03 13:25:53.509642 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:53.509606    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:53.511238 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:53.511209    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 13:25:53.511392 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:53.511276    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 13:25:53.831023 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:53.830932    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54" event={"ID":"0c76fdca-f0ac-418c-80df-623a97134eae","Type":"ContainerStarted","Data":"76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3"}
Nov 03 13:25:53.831183 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:53.831162    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:25:53.831229 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:53.831210    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54" event={"ID":"0c76fdca-f0ac-418c-80df-623a97134eae","Type":"ContainerStarted","Data":"5ec5a3519657efd8f6ab34ce7738af75bb532801195cd3801750ef6dc7ffc5ac"}
Nov 03 13:25:53.853367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:53.853038    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54" podStartSLOduration=1.294918067 podStartE2EDuration="1.853020097s" totalImagesPullingTime="558.10203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:53.849769118 +0000 UTC m=+1041.661437330" watchObservedRunningTime="2026-11-03 13:25:53.853020097 +0000 UTC m=+1041.664688286"
Nov 03 13:25:54.509727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:54.509691    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 13:25:54.510130 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:54.509754    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 13:25:55.509784 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:55.509744    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 13:25:55.510174 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:55.509810    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 13:25:56.510227 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:56.510152    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 13:25:56.510761 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:56.510274    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 13:25:56.513924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:56.513895    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"]
Nov 03 13:25:56.517159 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:56.517139    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:25:57.509288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:57.509256    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 13:25:57.509471 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:57.509312    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 13:25:58.509880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:58.509840    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 13:25:58.510266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:58.509902    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 13:25:58.795317 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:58.795232    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:25:59.513288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:59.513259    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:59.513708 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:59.513538    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:59.514385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:59.514369    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:25:59.864928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:25:59.864855    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:26:00.809323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:00.809292    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:26:01.514692 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.514644    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8" podUID="6763dd66-263e-4b5b-9f9e-57a269ca1493" containerName="payload-processing" containerID="cri-o://db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b" gracePeriod=25
Nov 03 13:26:01.806538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.806508    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:26:01.917826 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.917791    4447 generic.go:356] "Generic (PLEG): container finished" podID="6763dd66-263e-4b5b-9f9e-57a269ca1493" containerID="db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b" exitCode=0
Nov 03 13:26:01.918316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.917845    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"
Nov 03 13:26:01.918316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.917878    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8" event={"ID":"6763dd66-263e-4b5b-9f9e-57a269ca1493","Type":"ContainerDied","Data":"db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b"}
Nov 03 13:26:01.918316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.917946    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8" event={"ID":"6763dd66-263e-4b5b-9f9e-57a269ca1493","Type":"ContainerDied","Data":"f1c21b842736a9ade44c4d3590f86db4a8399ac0c4e045b42eb7108b9168f0ba"}
Nov 03 13:26:01.918316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.917981    4447 scope.go:122] "RemoveContainer" containerID="db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b"
Nov 03 13:26:01.928304 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.928278    4447 scope.go:122] "RemoveContainer" containerID="db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b"
Nov 03 13:26:01.928598 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:26:01.928579    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b\": container with ID starting with db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b not found: ID does not exist" containerID="db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b"
Nov 03 13:26:01.928651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.928608    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b"} err="failed to get container status \"db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b\": rpc error: code = NotFound desc = could not find container \"db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b\": container with ID starting with db2a8d606681ffdb141f242bad502cd24e323004dfef5e9f57b3529faf6c321b not found: ID does not exist"
Nov 03 13:26:01.952914 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.952885    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-provider-credentials\") pod \"6763dd66-263e-4b5b-9f9e-57a269ca1493\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") "
Nov 03 13:26:01.953123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.952980    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-kube-api-access-7gvhj\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-kube-api-access-7gvhj\") pod \"6763dd66-263e-4b5b-9f9e-57a269ca1493\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") "
Nov 03 13:26:01.953123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.953043    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-plugins-config-volume\") pod \"6763dd66-263e-4b5b-9f9e-57a269ca1493\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") "
Nov 03 13:26:01.953256 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.953123    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-routing-overlay-volume\") pod \"6763dd66-263e-4b5b-9f9e-57a269ca1493\" (UID: \"6763dd66-263e-4b5b-9f9e-57a269ca1493\") "
Nov 03 13:26:01.953571 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.953541    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-plugins-config-volume" pod "6763dd66-263e-4b5b-9f9e-57a269ca1493" (UID: "6763dd66-263e-4b5b-9f9e-57a269ca1493"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:01.953709 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.953635    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-routing-overlay-volume" pod "6763dd66-263e-4b5b-9f9e-57a269ca1493" (UID: "6763dd66-263e-4b5b-9f9e-57a269ca1493"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:01.953709 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.953699    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:01.955437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.955406    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-kube-api-access-7gvhj" pod "6763dd66-263e-4b5b-9f9e-57a269ca1493" (UID: "6763dd66-263e-4b5b-9f9e-57a269ca1493"). InnerVolumeSpecName "kube-api-access-7gvhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:01.955641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:01.955620    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-provider-credentials" pod "6763dd66-263e-4b5b-9f9e-57a269ca1493" (UID: "6763dd66-263e-4b5b-9f9e-57a269ca1493"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:02.055376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:02.055264    4447 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-provider-credentials\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:02.055376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:02.055296    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gvhj\" (UniqueName: \"kubernetes.io/projected/6763dd66-263e-4b5b-9f9e-57a269ca1493-kube-api-access-7gvhj\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:02.055376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:02.055306    4447 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6763dd66-263e-4b5b-9f9e-57a269ca1493-routing-overlay-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:02.242099 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:02.242036    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"]
Nov 03 13:26:02.245816 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:02.245786    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4bs59v8"]
Nov 03 13:26:02.324120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:02.324024    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6763dd66-263e-4b5b-9f9e-57a269ca1493" path="/var/lib/kubelet/pods/6763dd66-263e-4b5b-9f9e-57a269ca1493/volumes"
Nov 03 13:26:03.411406 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:26:03.411368    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:26:10.079299 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.079270    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"]
Nov 03 13:26:10.079900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.079883    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6763dd66-263e-4b5b-9f9e-57a269ca1493" containerName="payload-processing"
Nov 03 13:26:10.079968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.079901    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6763dd66-263e-4b5b-9f9e-57a269ca1493" containerName="payload-processing"
Nov 03 13:26:10.080037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.080023    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6763dd66-263e-4b5b-9f9e-57a269ca1493" containerName="payload-processing"
Nov 03 13:26:10.085029 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.085008    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.094706 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.094560    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"]
Nov 03 13:26:10.098660 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.098635    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8f8aa8\"/\"openshift-service-ca.crt\""
Nov 03 13:26:10.098969 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.098951    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8f8aa8\"/\"default-dockercfg-94nwg\""
Nov 03 13:26:10.099192 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.099176    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8f8aa8\"/\"kube-root-ca.crt\""
Nov 03 13:26:10.099400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.099381    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8f8aa8\"/\"auth-test-model-8f8aa8-kserve-self-signed-certs\""
Nov 03 13:26:10.267187 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.267145    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-tmp-dir\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.267508 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.267484    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-model-cache\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.267746 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.267728    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-home\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.267868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.267853    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kvh\" (UniqueName: \"kubernetes.io/projected/f0562f99-ac88-442d-b17b-86deec063aff-kube-api-access-h8kvh\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.268003 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.267988    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-dshm\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.268162 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.268148    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0562f99-ac88-442d-b17b-86deec063aff-tls-certs\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.373022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.369633    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-home\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.373022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.369696    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kvh\" (UniqueName: \"kubernetes.io/projected/f0562f99-ac88-442d-b17b-86deec063aff-kube-api-access-h8kvh\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.373022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.369755    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-dshm\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.373022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.369842    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0562f99-ac88-442d-b17b-86deec063aff-tls-certs\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.373022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.369908    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-tmp-dir\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.373022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.370011    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-model-cache\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.373022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.370572    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-model-cache\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.373022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.370957    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-home\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.373022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.371912    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-tmp-dir\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.383403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.383378    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-dshm\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.394981 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.394951    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kvh\" (UniqueName: \"kubernetes.io/projected/f0562f99-ac88-442d-b17b-86deec063aff-kube-api-access-h8kvh\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.412986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.412923    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0562f99-ac88-442d-b17b-86deec063aff-tls-certs\") pod \"auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:10.711662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:10.711624    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:11.357415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:11.357384    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"]
Nov 03 13:26:11.358074 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:26:11.358025    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0562f99_ac88_442d_b17b_86deec063aff.slice/crio-c3c982cb7417cc8983b06c14858cc1faa3b28708a0b63ac6f029a3256b9af11e WatchSource:0}: Error finding container c3c982cb7417cc8983b06c14858cc1faa3b28708a0b63ac6f029a3256b9af11e: Status 404 returned error can't find the container with id c3c982cb7417cc8983b06c14858cc1faa3b28708a0b63ac6f029a3256b9af11e
Nov 03 13:26:12.024311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:12.024274    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh" event={"ID":"f0562f99-ac88-442d-b17b-86deec063aff","Type":"ContainerStarted","Data":"01417c1712addc54801409abb1a8513ab7b1386486bc9415ad7cbbde1f8130c4"}
Nov 03 13:26:12.024571 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:12.024551    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh" event={"ID":"f0562f99-ac88-442d-b17b-86deec063aff","Type":"ContainerStarted","Data":"c3c982cb7417cc8983b06c14858cc1faa3b28708a0b63ac6f029a3256b9af11e"}
Nov 03 13:26:12.047406 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:12.047329    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh" podStartSLOduration=2.047310237 podStartE2EDuration="2.047310237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:12.042109351 +0000 UTC m=+1059.853777566" watchObservedRunningTime="2026-11-03 13:26:12.047310237 +0000 UTC m=+1059.858978428"
Nov 03 13:26:12.712926 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:12.712890    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:14.022025 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:14.021995    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"]
Nov 03 13:26:14.025117 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:14.025092    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6540d27d-8ccj8"]
Nov 03 13:26:14.321465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:14.321380    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3271d72-1127-4a45-bff1-23b7c80b9c86" path="/var/lib/kubelet/pods/b3271d72-1127-4a45-bff1-23b7c80b9c86/volumes"
Nov 03 13:26:18.414683 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:18.414646    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"]
Nov 03 13:26:19.172627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.163462    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"]
Nov 03 13:26:19.172958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.172922    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54" podUID="0c76fdca-f0ac-418c-80df-623a97134eae" containerName="maas-api" containerID="cri-o://76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3" gracePeriod=30
Nov 03 13:26:19.509837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.509806    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:26:19.551796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.545922    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c76fdca-f0ac-418c-80df-623a97134eae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c76fdca-f0ac-418c-80df-623a97134eae-openshift-service-ca\") pod \"0c76fdca-f0ac-418c-80df-623a97134eae\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") "
Nov 03 13:26:19.551796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.546037    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c76fdca-f0ac-418c-80df-623a97134eae-kube-api-access-lhrgd\" (UniqueName: \"kubernetes.io/projected/0c76fdca-f0ac-418c-80df-623a97134eae-kube-api-access-lhrgd\") pod \"0c76fdca-f0ac-418c-80df-623a97134eae\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") "
Nov 03 13:26:19.551796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.546123    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-maas-api-tls\") pod \"0c76fdca-f0ac-418c-80df-623a97134eae\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") "
Nov 03 13:26:19.551796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.546237    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-metrics-certs\") pod \"0c76fdca-f0ac-418c-80df-623a97134eae\" (UID: \"0c76fdca-f0ac-418c-80df-623a97134eae\") "
Nov 03 13:26:19.553424 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.553387    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c76fdca-f0ac-418c-80df-623a97134eae-openshift-service-ca" pod "0c76fdca-f0ac-418c-80df-623a97134eae" (UID: "0c76fdca-f0ac-418c-80df-623a97134eae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:19.555172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.555137    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-metrics-certs" pod "0c76fdca-f0ac-418c-80df-623a97134eae" (UID: "0c76fdca-f0ac-418c-80df-623a97134eae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:19.559760 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.559735    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c76fdca-f0ac-418c-80df-623a97134eae-kube-api-access-lhrgd" pod "0c76fdca-f0ac-418c-80df-623a97134eae" (UID: "0c76fdca-f0ac-418c-80df-623a97134eae"). InnerVolumeSpecName "kube-api-access-lhrgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:19.561137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.561111    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-maas-api-tls" pod "0c76fdca-f0ac-418c-80df-623a97134eae" (UID: "0c76fdca-f0ac-418c-80df-623a97134eae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:19.655408 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.654521    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:19.655408 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.654555    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c76fdca-f0ac-418c-80df-623a97134eae-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:19.655408 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.654571    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c76fdca-f0ac-418c-80df-623a97134eae-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:19.655408 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:19.654585    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrgd\" (UniqueName: \"kubernetes.io/projected/0c76fdca-f0ac-418c-80df-623a97134eae-kube-api-access-lhrgd\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:20.148893 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.148861    4447 generic.go:356] "Generic (PLEG): container finished" podID="0c76fdca-f0ac-418c-80df-623a97134eae" containerID="76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3" exitCode=0
Nov 03 13:26:20.149077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.148909    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"
Nov 03 13:26:20.149077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.148942    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54" event={"ID":"0c76fdca-f0ac-418c-80df-623a97134eae","Type":"ContainerDied","Data":"76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3"}
Nov 03 13:26:20.149077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.149001    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54" event={"ID":"0c76fdca-f0ac-418c-80df-623a97134eae","Type":"ContainerDied","Data":"5ec5a3519657efd8f6ab34ce7738af75bb532801195cd3801750ef6dc7ffc5ac"}
Nov 03 13:26:20.149077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.149031    4447 scope.go:122] "RemoveContainer" containerID="76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3"
Nov 03 13:26:20.160298 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.160278    4447 scope.go:122] "RemoveContainer" containerID="76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3"
Nov 03 13:26:20.160612 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:26:20.160586    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3\": container with ID starting with 76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3 not found: ID does not exist" containerID="76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3"
Nov 03 13:26:20.160720 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.160619    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3"} err="failed to get container status \"76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3\": rpc error: code = NotFound desc = could not find container \"76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3\": container with ID starting with 76d0f431978c45da6df6119872f3a9365c33da5712fd975f732848972dd27ab3 not found: ID does not exist"
Nov 03 13:26:20.172768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.172742    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"]
Nov 03 13:26:20.181811 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.181789    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dbf739c-6476f8cccb-p6p54"]
Nov 03 13:26:20.321018 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.320986    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c76fdca-f0ac-418c-80df-623a97134eae" path="/var/lib/kubelet/pods/0c76fdca-f0ac-418c-80df-623a97134eae/volumes"
Nov 03 13:26:20.712893 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.712860    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:20.732561 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:20.732046    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:21.171427 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:21.170171    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:26:23.415292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:23.415235    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd" podUID="50eaedc3-94f7-478c-b0ac-aac35df2b25e" containerName="payload-pre-processing" containerID="cri-o://7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36" gracePeriod=25
Nov 03 13:26:23.700044 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:23.700015    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:26:23.817207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:23.817172    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50eaedc3-94f7-478c-b0ac-aac35df2b25e-kube-api-access-zxctm\" (UniqueName: \"kubernetes.io/projected/50eaedc3-94f7-478c-b0ac-aac35df2b25e-kube-api-access-zxctm\") pod \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\" (UID: \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\") "
Nov 03 13:26:23.817388 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:23.817276    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50eaedc3-94f7-478c-b0ac-aac35df2b25e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaedc3-94f7-478c-b0ac-aac35df2b25e-plugins-config-volume\") pod \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\" (UID: \"50eaedc3-94f7-478c-b0ac-aac35df2b25e\") "
Nov 03 13:26:23.817729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:23.817704    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50eaedc3-94f7-478c-b0ac-aac35df2b25e-plugins-config-volume" pod "50eaedc3-94f7-478c-b0ac-aac35df2b25e" (UID: "50eaedc3-94f7-478c-b0ac-aac35df2b25e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:23.817958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:23.817943    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaedc3-94f7-478c-b0ac-aac35df2b25e-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:23.819675 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:23.819645    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eaedc3-94f7-478c-b0ac-aac35df2b25e-kube-api-access-zxctm" pod "50eaedc3-94f7-478c-b0ac-aac35df2b25e" (UID: "50eaedc3-94f7-478c-b0ac-aac35df2b25e"). InnerVolumeSpecName "kube-api-access-zxctm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:23.919467 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:23.919431    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxctm\" (UniqueName: \"kubernetes.io/projected/50eaedc3-94f7-478c-b0ac-aac35df2b25e-kube-api-access-zxctm\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:24.178845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.178810    4447 generic.go:356] "Generic (PLEG): container finished" podID="50eaedc3-94f7-478c-b0ac-aac35df2b25e" containerID="7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36" exitCode=0
Nov 03 13:26:24.179036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.178859    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"
Nov 03 13:26:24.179036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.178890    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd" event={"ID":"50eaedc3-94f7-478c-b0ac-aac35df2b25e","Type":"ContainerDied","Data":"7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36"}
Nov 03 13:26:24.179036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.178962    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd" event={"ID":"50eaedc3-94f7-478c-b0ac-aac35df2b25e","Type":"ContainerDied","Data":"d1471c982e06ecc0aa1ca50019af0d70dd2e8aceb1c0a697594e94bee9fefe54"}
Nov 03 13:26:24.179036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.178995    4447 scope.go:122] "RemoveContainer" containerID="7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36"
Nov 03 13:26:24.190910 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.190893    4447 scope.go:122] "RemoveContainer" containerID="7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36"
Nov 03 13:26:24.191196 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:26:24.191175    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36\": container with ID starting with 7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36 not found: ID does not exist" containerID="7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36"
Nov 03 13:26:24.191243 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.191205    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36"} err="failed to get container status \"7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36\": rpc error: code = NotFound desc = could not find container \"7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36\": container with ID starting with 7c703dd5d4e907f9983f5da73c16ab12e3b48df0eeb65153f497e73859eafc36 not found: ID does not exist"
Nov 03 13:26:24.207681 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.207653    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"]
Nov 03 13:26:24.213670 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.213647    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2dbf739c-6976fdf8c9-f4jpd"]
Nov 03 13:26:24.320030 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.319996    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50eaedc3-94f7-478c-b0ac-aac35df2b25e" path="/var/lib/kubelet/pods/50eaedc3-94f7-478c-b0ac-aac35df2b25e/volumes"
Nov 03 13:26:24.714623 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.714590    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"]
Nov 03 13:26:24.715150 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.715135    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0c76fdca-f0ac-418c-80df-623a97134eae" containerName="maas-api"
Nov 03 13:26:24.715194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.715151    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c76fdca-f0ac-418c-80df-623a97134eae" containerName="maas-api"
Nov 03 13:26:24.715194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.715188    4447 cpu_manager.go:414] "containerMap: removing container" podUID="50eaedc3-94f7-478c-b0ac-aac35df2b25e" containerName="payload-pre-processing"
Nov 03 13:26:24.715263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.715194    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50eaedc3-94f7-478c-b0ac-aac35df2b25e" containerName="payload-pre-processing"
Nov 03 13:26:24.715305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.715274    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="50eaedc3-94f7-478c-b0ac-aac35df2b25e" containerName="payload-pre-processing"
Nov 03 13:26:24.715305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.715285    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c76fdca-f0ac-418c-80df-623a97134eae" containerName="maas-api"
Nov 03 13:26:24.721385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.720324    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:24.727986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.727951    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e04a4e0c\""
Nov 03 13:26:24.727986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.727964    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e04a4e0c-dockercfg-b5xkp\""
Nov 03 13:26:24.728928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.728903    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"]
Nov 03 13:26:24.735539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.735519    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"]
Nov 03 13:26:24.735785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.735735    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:24.751822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.751179    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"]
Nov 03 13:26:24.839573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.839179    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afa6f57-2302-47ea-8e8a-37f107368b64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc\" (UID: \"4afa6f57-2302-47ea-8e8a-37f107368b64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:24.839991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.839964    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9992\" (UniqueName: \"kubernetes.io/projected/4afa6f57-2302-47ea-8e8a-37f107368b64-kube-api-access-f9992\") pod \"payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc\" (UID: \"4afa6f57-2302-47ea-8e8a-37f107368b64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:24.840365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.840320    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ed5d27-cfea-4f4d-9772-384fea8a319b-plugins-config-volume\") pod \"payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4\" (UID: \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\") " pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:24.840693 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.840671    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrwvm\" (UniqueName: \"kubernetes.io/projected/f0ed5d27-cfea-4f4d-9772-384fea8a319b-kube-api-access-zrwvm\") pod \"payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4\" (UID: \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\") " pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:24.962371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.957364    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afa6f57-2302-47ea-8e8a-37f107368b64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc\" (UID: \"4afa6f57-2302-47ea-8e8a-37f107368b64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:24.962371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.957819    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9992\" (UniqueName: \"kubernetes.io/projected/4afa6f57-2302-47ea-8e8a-37f107368b64-kube-api-access-f9992\") pod \"payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc\" (UID: \"4afa6f57-2302-47ea-8e8a-37f107368b64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:24.962371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.957988    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ed5d27-cfea-4f4d-9772-384fea8a319b-plugins-config-volume\") pod \"payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4\" (UID: \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\") " pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:24.962371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.958248    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrwvm\" (UniqueName: \"kubernetes.io/projected/f0ed5d27-cfea-4f4d-9772-384fea8a319b-kube-api-access-zrwvm\") pod \"payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4\" (UID: \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\") " pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:24.962371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.959686    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afa6f57-2302-47ea-8e8a-37f107368b64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc\" (UID: \"4afa6f57-2302-47ea-8e8a-37f107368b64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:24.962371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.960203    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ed5d27-cfea-4f4d-9772-384fea8a319b-plugins-config-volume\") pod \"payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4\" (UID: \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\") " pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:24.973044 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.972969    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9992\" (UniqueName: \"kubernetes.io/projected/4afa6f57-2302-47ea-8e8a-37f107368b64-kube-api-access-f9992\") pod \"payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc\" (UID: \"4afa6f57-2302-47ea-8e8a-37f107368b64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:24.973259 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:24.972971    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrwvm\" (UniqueName: \"kubernetes.io/projected/f0ed5d27-cfea-4f4d-9772-384fea8a319b-kube-api-access-zrwvm\") pod \"payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4\" (UID: \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\") " pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:25.041727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:25.041692    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:25.064297 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:25.064245    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:25.409212 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:25.409178    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"]
Nov 03 13:26:25.410197 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:26:25.410172    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ed5d27_cfea_4f4d_9772_384fea8a319b.slice/crio-3959eb354e0268863f03e45615717a538665ad1def30e14e65e499c8045594bb WatchSource:0}: Error finding container 3959eb354e0268863f03e45615717a538665ad1def30e14e65e499c8045594bb: Status 404 returned error can't find the container with id 3959eb354e0268863f03e45615717a538665ad1def30e14e65e499c8045594bb
Nov 03 13:26:25.432609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:25.432583    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"]
Nov 03 13:26:25.433680 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:26:25.433651    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afa6f57_2302_47ea_8e8a_37f107368b64.slice/crio-6215f513e35bca1bb11169ab5927e25a6c48d2de4df645f50dc8773fef905c25 WatchSource:0}: Error finding container 6215f513e35bca1bb11169ab5927e25a6c48d2de4df645f50dc8773fef905c25: Status 404 returned error can't find the container with id 6215f513e35bca1bb11169ab5927e25a6c48d2de4df645f50dc8773fef905c25
Nov 03 13:26:26.203301 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:26.203254    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4" event={"ID":"f0ed5d27-cfea-4f4d-9772-384fea8a319b","Type":"ContainerStarted","Data":"ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7"}
Nov 03 13:26:26.203850 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:26.203447    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4" event={"ID":"f0ed5d27-cfea-4f4d-9772-384fea8a319b","Type":"ContainerStarted","Data":"3959eb354e0268863f03e45615717a538665ad1def30e14e65e499c8045594bb"}
Nov 03 13:26:26.203850 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:26.203823    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:26.205357 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:26.205312    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc" event={"ID":"4afa6f57-2302-47ea-8e8a-37f107368b64","Type":"ContainerStarted","Data":"56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a"}
Nov 03 13:26:26.205497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:26.205396    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc" event={"ID":"4afa6f57-2302-47ea-8e8a-37f107368b64","Type":"ContainerStarted","Data":"6215f513e35bca1bb11169ab5927e25a6c48d2de4df645f50dc8773fef905c25"}
Nov 03 13:26:26.205497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:26.205451    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:26.219809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:26.219747    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4" podStartSLOduration=2.219728423 podStartE2EDuration="2.219728423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:26.219055617 +0000 UTC m=+1074.030723806" watchObservedRunningTime="2026-11-03 13:26:26.219728423 +0000 UTC m=+1074.031396612"
Nov 03 13:26:26.237146 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:26.237073    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc" podStartSLOduration=2.237047538 podStartE2EDuration="2.237047538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:26.233214905 +0000 UTC m=+1074.044883094" watchObservedRunningTime="2026-11-03 13:26:26.237047538 +0000 UTC m=+1074.048715727"
Nov 03 13:26:28.135717 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.135679    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"]
Nov 03 13:26:28.140958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.140933    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.161015 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.160984    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"]
Nov 03 13:26:28.230197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.230168    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9add72-openshift-default-dockercfg-c6nrt\""
Nov 03 13:26:28.315297 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.315262    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaf70db-dc8c-451f-a584-77b665a67fd3-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.315490 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.315386    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.315490 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.315433    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.315576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.315494    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdlc\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-kube-api-access-kzdlc\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.315576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.315529    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.315644 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.315622    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.315686 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.315643    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.315686 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.315663    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.315755 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.315723    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.416585 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.416504    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaf70db-dc8c-451f-a584-77b665a67fd3-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.416761 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.416735    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.416854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.416832    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.416927 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.416867    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdlc\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-kube-api-access-kzdlc\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.416927 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.416901    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.417100 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.417079    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.417186 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.417122    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.417186 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.417168    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.417293 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.417269    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.417677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.417608    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.417807 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.417785    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.417852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.417819    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.417852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.417842    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.418248 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.418228    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaf70db-dc8c-451f-a584-77b665a67fd3-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.419267 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.419244    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.419876 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.419853    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.424569 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.424545    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.424920 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.424897    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdlc\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-kube-api-access-kzdlc\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.466562 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.466533    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:28.754963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.754928    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"]
Nov 03 13:26:28.756456 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:26:28.756419    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaaf70db_dc8c_451f_a584_77b665a67fd3.slice/crio-8250e1504bc1125e4360d8298bca3109f53aa4e6b18b45e0b9e0574bb7687ba2 WatchSource:0}: Error finding container 8250e1504bc1125e4360d8298bca3109f53aa4e6b18b45e0b9e0574bb7687ba2: Status 404 returned error can't find the container with id 8250e1504bc1125e4360d8298bca3109f53aa4e6b18b45e0b9e0574bb7687ba2
Nov 03 13:26:28.761744 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.760376    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:26:28.761744 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.760555    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:26:28.761744 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:28.760602    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:26:29.228429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:29.228385    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" event={"ID":"aaaf70db-dc8c-451f-a584-77b665a67fd3","Type":"ContainerStarted","Data":"4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3"}
Nov 03 13:26:29.229475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:29.228459    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" event={"ID":"aaaf70db-dc8c-451f-a584-77b665a67fd3","Type":"ContainerStarted","Data":"8250e1504bc1125e4360d8298bca3109f53aa4e6b18b45e0b9e0574bb7687ba2"}
Nov 03 13:26:29.251502 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:29.251446    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" podStartSLOduration=1.251431958 podStartE2EDuration="1.251431958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:29.247796598 +0000 UTC m=+1077.059464835" watchObservedRunningTime="2026-11-03 13:26:29.251431958 +0000 UTC m=+1077.063100145"
Nov 03 13:26:29.467546 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:29.467507    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:29.469569 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:29.469539    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 13:26:29.469785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:29.469757    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 13:26:30.467263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:30.467225    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 13:26:30.467672 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:30.467283    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 13:26:31.391499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.391467    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"]
Nov 03 13:26:31.401209 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.401183    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"]
Nov 03 13:26:31.401390 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.401302    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.407630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.407596    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e04a4e0c-serving-cert\""
Nov 03 13:26:31.470942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.469477    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 13:26:31.470942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.469535    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 13:26:31.487385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.486282    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvl5\" (UniqueName: \"kubernetes.io/projected/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-kube-api-access-7hvl5\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.487385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.486335    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-openshift-service-ca\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.487385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.486428    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-maas-api-tls\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.487385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.486672    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-metrics-certs\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.588620 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.588579    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-metrics-certs\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.588817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.588660    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvl5\" (UniqueName: \"kubernetes.io/projected/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-kube-api-access-7hvl5\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.588817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.588691    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-openshift-service-ca\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.589599 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.589575    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-openshift-service-ca\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.589705 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.589638    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-maas-api-tls\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.591788 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.591762    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-metrics-certs\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.592704 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.592674    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-maas-api-tls\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.596125 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.596100    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvl5\" (UniqueName: \"kubernetes.io/projected/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-kube-api-access-7hvl5\") pod \"maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:31.721142 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:31.721109    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:32.061756 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:32.056318    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"]
Nov 03 13:26:32.084413 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:26:32.077293    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a78eafb_0ec3_41bf_819a_cdfb29dc1dfe.slice/crio-922ec3b6957133a47834f8841b73c22335517c51d5bd8e405d5e7a8608002b04 WatchSource:0}: Error finding container 922ec3b6957133a47834f8841b73c22335517c51d5bd8e405d5e7a8608002b04: Status 404 returned error can't find the container with id 922ec3b6957133a47834f8841b73c22335517c51d5bd8e405d5e7a8608002b04
Nov 03 13:26:32.277934 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:32.277899    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr" event={"ID":"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe","Type":"ContainerStarted","Data":"922ec3b6957133a47834f8841b73c22335517c51d5bd8e405d5e7a8608002b04"}
Nov 03 13:26:32.467097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:32.467067    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 13:26:32.467266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:32.467116    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 13:26:33.288891 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:33.288848    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr" event={"ID":"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe","Type":"ContainerStarted","Data":"0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf"}
Nov 03 13:26:33.289310 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:33.288976    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:33.307798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:33.307740    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr" podStartSLOduration=1.840225897 podStartE2EDuration="2.30772562s" totalImagesPullingTime="467.499723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:33.306261096 +0000 UTC m=+1081.117929321" watchObservedRunningTime="2026-11-03 13:26:33.30772562 +0000 UTC m=+1081.119393808"
Nov 03 13:26:33.413284 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:26:33.413247    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:26:33.473367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:33.470681    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 13:26:33.473367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:33.470743    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 13:26:34.467207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:34.467173    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 13:26:34.467650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:34.467244    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 13:26:35.471315 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:35.471280    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:35.471799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:35.471600    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:35.472520 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:35.472496    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:26:37.216169 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:37.216131    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:26:37.216676 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:37.216258    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:26:39.302456 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:39.302426    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:47.998457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:47.998422    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"]
Nov 03 13:26:48.003802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.003763    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.007030 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.007002    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-8f8aa8-openshift-default-dockercfg-cvr4j\""
Nov 03 13:26:48.036804 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.036762    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"]
Nov 03 13:26:48.048382 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.048324    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.048550 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.048508    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.048621 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.048554    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.048684 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.048651    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.048740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.048693    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jcs\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-kube-api-access-x9jcs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.048803 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.048757    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.048859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.048800    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.048859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.048834    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.048964 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.048906    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.150441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150409    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.150441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150453    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.150694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150495    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.150694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150516    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jcs\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-kube-api-access-x9jcs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.150694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150562    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.150694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150604    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.150694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150635    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.151027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150700    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.151027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150814    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.151027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150894    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.151027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.150921    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.151284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.151157    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.151492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.151396    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.151492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.151408    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.153421 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.153397    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.153604 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.153583    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.157784 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.157762    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.158159 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.158130    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jcs\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-kube-api-access-x9jcs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.342261 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.342179    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:48.639614 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:26:48.639564    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddc3b97_c217_4a76_97cc_c0d6fab1c400.slice/crio-a3a8d9ffb02420a4a40bf4977140ba14366963c2865dd4706df58d8c6499928a WatchSource:0}: Error finding container a3a8d9ffb02420a4a40bf4977140ba14366963c2865dd4706df58d8c6499928a: Status 404 returned error can't find the container with id a3a8d9ffb02420a4a40bf4977140ba14366963c2865dd4706df58d8c6499928a
Nov 03 13:26:48.639818 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.639787    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"]
Nov 03 13:26:48.642973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.642936    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:26:48.643134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.643023    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:26:48.643134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:48.643066    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:26:49.400040 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:49.400002    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" event={"ID":"4ddc3b97-c217-4a76-97cc-c0d6fab1c400","Type":"ContainerStarted","Data":"961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8"}
Nov 03 13:26:49.400525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:49.400084    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" event={"ID":"4ddc3b97-c217-4a76-97cc-c0d6fab1c400","Type":"ContainerStarted","Data":"a3a8d9ffb02420a4a40bf4977140ba14366963c2865dd4706df58d8c6499928a"}
Nov 03 13:26:49.421625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:49.421576    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" podStartSLOduration=2.421562935 podStartE2EDuration="2.421562935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:49.420102544 +0000 UTC m=+1097.231770753" watchObservedRunningTime="2026-11-03 13:26:49.421562935 +0000 UTC m=+1097.233231122"
Nov 03 13:26:50.348139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:50.345697    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:50.360454 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:50.360040    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 13:26:50.360454 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:50.360113    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 13:26:51.356368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:51.355771    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 13:26:51.356368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:51.355836    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 13:26:52.355367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:52.350783    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 13:26:52.355367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:52.350845    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 13:26:53.357365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:53.356835    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 13:26:53.357365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:53.356919    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 13:26:54.357367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:54.356726    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 13:26:54.357367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:54.356799    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 13:26:54.767684 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:54.767639    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"]
Nov 03 13:26:54.784019 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:54.783983    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"]
Nov 03 13:26:55.361885 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:55.361852    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:55.362334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:55.362161    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:55.363086 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:55.363062    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:26:56.957856 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:56.957823    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"]
Nov 03 13:26:56.958320 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:56.958065    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr" podUID="0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" containerName="maas-api" containerID="cri-o://0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf" gracePeriod=30
Nov 03 13:26:57.435205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.435166    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:57.458050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.458014    4447 generic.go:356] "Generic (PLEG): container finished" podID="0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" containerID="0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf" exitCode=0
Nov 03 13:26:57.459553 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.458092    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"
Nov 03 13:26:57.459553 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.458089    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr" event={"ID":"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe","Type":"ContainerDied","Data":"0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf"}
Nov 03 13:26:57.459553 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.458147    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr" event={"ID":"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe","Type":"ContainerDied","Data":"922ec3b6957133a47834f8841b73c22335517c51d5bd8e405d5e7a8608002b04"}
Nov 03 13:26:57.459553 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.458179    4447 scope.go:122] "RemoveContainer" containerID="0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf"
Nov 03 13:26:57.494564 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.494425    4447 scope.go:122] "RemoveContainer" containerID="0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf"
Nov 03 13:26:57.494870 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:26:57.494840    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf\": container with ID starting with 0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf not found: ID does not exist" containerID="0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf"
Nov 03 13:26:57.494973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.494883    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf"} err="failed to get container status \"0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf\": rpc error: code = NotFound desc = could not find container \"0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf\": container with ID starting with 0b8b1df8e5a836bd0d1679dd5be389a15e2bdb494b79303588e9fce1c18f2ddf not found: ID does not exist"
Nov 03 13:26:57.534869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.534836    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-openshift-service-ca\") pod \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") "
Nov 03 13:26:57.534869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.534877    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-metrics-certs\") pod \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") "
Nov 03 13:26:57.535141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.534933    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-maas-api-tls\") pod \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") "
Nov 03 13:26:57.535141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.534986    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-kube-api-access-7hvl5\" (UniqueName: \"kubernetes.io/projected/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-kube-api-access-7hvl5\") pod \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\" (UID: \"0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe\") "
Nov 03 13:26:57.535720 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.535685    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-openshift-service-ca" pod "0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" (UID: "0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:57.538163 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.538135    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-maas-api-tls" pod "0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" (UID: "0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:57.538396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.538336    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-metrics-certs" pod "0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" (UID: "0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:57.538494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.538447    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-kube-api-access-7hvl5" pod "0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" (UID: "0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe"). InnerVolumeSpecName "kube-api-access-7hvl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:57.636495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.636465    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:57.636495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.636494    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:57.636495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.636503    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:57.636735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.636512    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hvl5\" (UniqueName: \"kubernetes.io/projected/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe-kube-api-access-7hvl5\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:57.786711 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.786681    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"]
Nov 03 13:26:57.795090 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:57.793579    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e04a4e0c-7c589c5cdc-578rr"]
Nov 03 13:26:58.320097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:58.320065    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" path="/var/lib/kubelet/pods/0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe/volumes"
Nov 03 13:26:59.768487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:59.768418    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4" podUID="f0ed5d27-cfea-4f4d-9772-384fea8a319b" containerName="payload-processing" containerID="cri-o://ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7" gracePeriod=25
Nov 03 13:26:59.785287 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:26:59.785251    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc" podUID="4afa6f57-2302-47ea-8e8a-37f107368b64" containerName="payload-pre-processing" containerID="cri-o://56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a" gracePeriod=25
Nov 03 13:27:00.174123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.173864    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:27:00.228505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.226107    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ed5d27-cfea-4f4d-9772-384fea8a319b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ed5d27-cfea-4f4d-9772-384fea8a319b-plugins-config-volume\") pod \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\" (UID: \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\") "
Nov 03 13:27:00.228505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.226158    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ed5d27-cfea-4f4d-9772-384fea8a319b-kube-api-access-zrwvm\" (UniqueName: \"kubernetes.io/projected/f0ed5d27-cfea-4f4d-9772-384fea8a319b-kube-api-access-zrwvm\") pod \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\" (UID: \"f0ed5d27-cfea-4f4d-9772-384fea8a319b\") "
Nov 03 13:27:00.229216 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.229188    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ed5d27-cfea-4f4d-9772-384fea8a319b-plugins-config-volume" pod "f0ed5d27-cfea-4f4d-9772-384fea8a319b" (UID: "f0ed5d27-cfea-4f4d-9772-384fea8a319b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:00.232565 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.232520    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ed5d27-cfea-4f4d-9772-384fea8a319b-kube-api-access-zrwvm" pod "f0ed5d27-cfea-4f4d-9772-384fea8a319b" (UID: "f0ed5d27-cfea-4f4d-9772-384fea8a319b"). InnerVolumeSpecName "kube-api-access-zrwvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:00.304877 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.304850    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:27:00.415442 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.344296    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4afa6f57-2302-47ea-8e8a-37f107368b64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afa6f57-2302-47ea-8e8a-37f107368b64-plugins-config-volume\") pod \"4afa6f57-2302-47ea-8e8a-37f107368b64\" (UID: \"4afa6f57-2302-47ea-8e8a-37f107368b64\") "
Nov 03 13:27:00.415442 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.344410    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4afa6f57-2302-47ea-8e8a-37f107368b64-kube-api-access-f9992\" (UniqueName: \"kubernetes.io/projected/4afa6f57-2302-47ea-8e8a-37f107368b64-kube-api-access-f9992\") pod \"4afa6f57-2302-47ea-8e8a-37f107368b64\" (UID: \"4afa6f57-2302-47ea-8e8a-37f107368b64\") "
Nov 03 13:27:00.415442 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.352914    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afa6f57-2302-47ea-8e8a-37f107368b64-plugins-config-volume" pod "4afa6f57-2302-47ea-8e8a-37f107368b64" (UID: "4afa6f57-2302-47ea-8e8a-37f107368b64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:00.416395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.416040    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afa6f57-2302-47ea-8e8a-37f107368b64-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:00.416395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.416073    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ed5d27-cfea-4f4d-9772-384fea8a319b-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:00.416395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.416088    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrwvm\" (UniqueName: \"kubernetes.io/projected/f0ed5d27-cfea-4f4d-9772-384fea8a319b-kube-api-access-zrwvm\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:00.498517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.498481    4447 generic.go:356] "Generic (PLEG): container finished" podID="4afa6f57-2302-47ea-8e8a-37f107368b64" containerID="56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a" exitCode=0
Nov 03 13:27:00.498752 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.498732    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"
Nov 03 13:27:00.499337 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.498742    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc" event={"ID":"4afa6f57-2302-47ea-8e8a-37f107368b64","Type":"ContainerDied","Data":"56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a"}
Nov 03 13:27:00.499551 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.499527    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc" event={"ID":"4afa6f57-2302-47ea-8e8a-37f107368b64","Type":"ContainerDied","Data":"6215f513e35bca1bb11169ab5927e25a6c48d2de4df645f50dc8773fef905c25"}
Nov 03 13:27:00.499648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.499571    4447 scope.go:122] "RemoveContainer" containerID="56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a"
Nov 03 13:27:00.501199 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.501178    4447 generic.go:356] "Generic (PLEG): container finished" podID="f0ed5d27-cfea-4f4d-9772-384fea8a319b" containerID="ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7" exitCode=0
Nov 03 13:27:00.501293 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.501245    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4" event={"ID":"f0ed5d27-cfea-4f4d-9772-384fea8a319b","Type":"ContainerDied","Data":"ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7"}
Nov 03 13:27:00.501293 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.501284    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4" event={"ID":"f0ed5d27-cfea-4f4d-9772-384fea8a319b","Type":"ContainerDied","Data":"3959eb354e0268863f03e45615717a538665ad1def30e14e65e499c8045594bb"}
Nov 03 13:27:00.501602 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.501587    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"
Nov 03 13:27:00.529681 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.528670    4447 scope.go:122] "RemoveContainer" containerID="56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a"
Nov 03 13:27:00.529681 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.528862    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afa6f57-2302-47ea-8e8a-37f107368b64-kube-api-access-f9992" pod "4afa6f57-2302-47ea-8e8a-37f107368b64" (UID: "4afa6f57-2302-47ea-8e8a-37f107368b64"). InnerVolumeSpecName "kube-api-access-f9992". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:00.529681 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.529588    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9992\" (UniqueName: \"kubernetes.io/projected/4afa6f57-2302-47ea-8e8a-37f107368b64-kube-api-access-f9992\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:00.529973 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:27:00.529717    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a\": container with ID starting with 56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a not found: ID does not exist" containerID="56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a"
Nov 03 13:27:00.529973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.529767    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a"} err="failed to get container status \"56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a\": rpc error: code = NotFound desc = could not find container \"56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a\": container with ID starting with 56b28ae5506024cedf0cd1e37897ac1ac837351890a8082db8a8ebe21ef82c2a not found: ID does not exist"
Nov 03 13:27:00.529973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.529791    4447 scope.go:122] "RemoveContainer" containerID="ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7"
Nov 03 13:27:00.554000 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.553138    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"]
Nov 03 13:27:00.593328 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.592215    4447 scope.go:122] "RemoveContainer" containerID="ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7"
Nov 03 13:27:00.594021 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:27:00.593868    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7\": container with ID starting with ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7 not found: ID does not exist" containerID="ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7"
Nov 03 13:27:00.594021 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.593941    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7"} err="failed to get container status \"ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7\": rpc error: code = NotFound desc = could not find container \"ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7\": container with ID starting with ee8e8d805efaa7e0cf203417178118f7de71916ad56f497e86f313d2040e31a7 not found: ID does not exist"
Nov 03 13:27:00.609368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.608939    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e04a4e0c-5645d65955-7g5s4"]
Nov 03 13:27:00.835629 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.835591    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"]
Nov 03 13:27:00.841104 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:00.841071    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e04a4e0c-6c7f8c7fc9-jx5xc"]
Nov 03 13:27:02.321501 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:02.321468    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4afa6f57-2302-47ea-8e8a-37f107368b64" path="/var/lib/kubelet/pods/4afa6f57-2302-47ea-8e8a-37f107368b64/volumes"
Nov 03 13:27:02.322119 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:02.322099    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ed5d27-cfea-4f4d-9772-384fea8a319b" path="/var/lib/kubelet/pods/f0ed5d27-cfea-4f4d-9772-384fea8a319b/volumes"
Nov 03 13:27:03.414369 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:27:03.414318    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:27:05.032502 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.032465    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"]
Nov 03 13:27:05.033098 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.033083    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" containerName="maas-api"
Nov 03 13:27:05.033098 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.033099    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" containerName="maas-api"
Nov 03 13:27:05.033272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.033127    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4afa6f57-2302-47ea-8e8a-37f107368b64" containerName="payload-pre-processing"
Nov 03 13:27:05.033272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.033135    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afa6f57-2302-47ea-8e8a-37f107368b64" containerName="payload-pre-processing"
Nov 03 13:27:05.033272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.033154    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f0ed5d27-cfea-4f4d-9772-384fea8a319b" containerName="payload-processing"
Nov 03 13:27:05.033272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.033159    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ed5d27-cfea-4f4d-9772-384fea8a319b" containerName="payload-processing"
Nov 03 13:27:05.033272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.033264    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afa6f57-2302-47ea-8e8a-37f107368b64" containerName="payload-pre-processing"
Nov 03 13:27:05.033272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.033274    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a78eafb-0ec3-41bf-819a-cdfb29dc1dfe" containerName="maas-api"
Nov 03 13:27:05.033511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.033281    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ed5d27-cfea-4f4d-9772-384fea8a319b" containerName="payload-processing"
Nov 03 13:27:05.038281 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.038263    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:05.051206 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.051180    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"]
Nov 03 13:27:05.051531 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.051508    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c87701ad\""
Nov 03 13:27:05.051977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.051954    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c87701ad-dockercfg-7vwzl\""
Nov 03 13:27:05.069463 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.069433    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"]
Nov 03 13:27:05.073620 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.073598    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:05.093286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.093241    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"]
Nov 03 13:27:05.118469 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.118439    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbvv\" (UniqueName: \"kubernetes.io/projected/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-kube-api-access-krbvv\") pod \"payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq\" (UID: \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:05.118632 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.118519    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq\" (UID: \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:05.223129 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.223089    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-plugins-config-volume\") pod \"payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz\" (UID: \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\") " pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:05.223335 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.223162    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbvv\" (UniqueName: \"kubernetes.io/projected/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-kube-api-access-krbvv\") pod \"payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq\" (UID: \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:05.223335 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.223225    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq\" (UID: \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:05.223335 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.223276    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrfd\" (UniqueName: \"kubernetes.io/projected/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-kube-api-access-flrfd\") pod \"payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz\" (UID: \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\") " pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:05.224408 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.224326    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq\" (UID: \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:05.251237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.251204    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbvv\" (UniqueName: \"kubernetes.io/projected/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-kube-api-access-krbvv\") pod \"payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq\" (UID: \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:05.324876 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.324781    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-plugins-config-volume\") pod \"payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz\" (UID: \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\") " pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:05.325036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.324969    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrfd\" (UniqueName: \"kubernetes.io/projected/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-kube-api-access-flrfd\") pod \"payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz\" (UID: \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\") " pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:05.325825 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.325776    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-plugins-config-volume\") pod \"payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz\" (UID: \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\") " pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:05.333280 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.333256    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrfd\" (UniqueName: \"kubernetes.io/projected/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-kube-api-access-flrfd\") pod \"payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz\" (UID: \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\") " pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:05.353582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.353546    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:05.393000 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.387869    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:05.653078 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.653035    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"]
Nov 03 13:27:05.654727 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:27:05.654705    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bb861f_b035_4ffa_9ce3_2eb5be3c5e4f.slice/crio-26be07ba6afd7942cb609af0c750a788d19827b505b64128c0e483f13996f85b WatchSource:0}: Error finding container 26be07ba6afd7942cb609af0c750a788d19827b505b64128c0e483f13996f85b: Status 404 returned error can't find the container with id 26be07ba6afd7942cb609af0c750a788d19827b505b64128c0e483f13996f85b
Nov 03 13:27:05.701647 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:05.700735    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"]
Nov 03 13:27:05.702312 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:27:05.702282    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cda5853_4fc9_41ce_8eaa_78cfa1cd2e63.slice/crio-7af098cda58e1bb40f29eaa1152cfc40fc0656f5a364278a6cd8d9655d67051e WatchSource:0}: Error finding container 7af098cda58e1bb40f29eaa1152cfc40fc0656f5a364278a6cd8d9655d67051e: Status 404 returned error can't find the container with id 7af098cda58e1bb40f29eaa1152cfc40fc0656f5a364278a6cd8d9655d67051e
Nov 03 13:27:06.560701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:06.560663    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz" event={"ID":"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63","Type":"ContainerStarted","Data":"62e618f78efdf3f856eed93448c73fa7c53895c946c25d46b17af683d329c216"}
Nov 03 13:27:06.561184 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:06.560725    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz" event={"ID":"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63","Type":"ContainerStarted","Data":"7af098cda58e1bb40f29eaa1152cfc40fc0656f5a364278a6cd8d9655d67051e"}
Nov 03 13:27:06.561184 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:06.560754    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:06.562284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:06.562264    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq" event={"ID":"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f","Type":"ContainerStarted","Data":"7b74d061973675d2b994432a2f0f86dafe73f20bc85e465d70a00dae92c7939c"}
Nov 03 13:27:06.562352 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:06.562303    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq" event={"ID":"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f","Type":"ContainerStarted","Data":"26be07ba6afd7942cb609af0c750a788d19827b505b64128c0e483f13996f85b"}
Nov 03 13:27:06.562352 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:06.562327    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:06.582626 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:06.582571    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz" podStartSLOduration=1.5825529409999999 podStartE2EDuration="1.582552941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:06.57863926 +0000 UTC m=+1114.390307471" watchObservedRunningTime="2026-11-03 13:27:06.582552941 +0000 UTC m=+1114.394221130"
Nov 03 13:27:06.598098 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:06.598047    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq" podStartSLOduration=1.598032685 podStartE2EDuration="1.598032685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:06.596787435 +0000 UTC m=+1114.408455653" watchObservedRunningTime="2026-11-03 13:27:06.598032685 +0000 UTC m=+1114.409700874"
Nov 03 13:27:10.991321 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:10.991285    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"]
Nov 03 13:27:10.997838 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:10.997812    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.009117 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.009089    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c87701ad-serving-cert\""
Nov 03 13:27:11.010486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.010460    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-openshift-service-ca\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.010733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.010711    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.010950 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.010933    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x96d\" (UniqueName: \"kubernetes.io/projected/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-kube-api-access-8x96d\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.011106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.011092    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-metrics-certs\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.015030 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.015008    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"]
Nov 03 13:27:11.113159 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.113077    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-metrics-certs\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.113466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.113195    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-openshift-service-ca\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.113466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.113265    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.113466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.113406    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x96d\" (UniqueName: \"kubernetes.io/projected/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-kube-api-access-8x96d\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.114785 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:27:11.114745    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-serving-cert: secret "maas-api-e2e-ait-c87701ad-serving-cert" not found
Nov 03 13:27:11.114939 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:27:11.114820    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls podName:4a7ba06d-1b55-4b6f-a1d3-500aca77c625 nodeName:}" failed. No retries permitted until 2026-11-03 13:27:11.614800712 +0000 UTC m=+1119.426468905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls") pod "maas-api-e2e-ait-c87701ad-5694955448-mp4mv" (UID: "4a7ba06d-1b55-4b6f-a1d3-500aca77c625") : secret "maas-api-e2e-ait-c87701ad-serving-cert" not found
Nov 03 13:27:11.114939 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.114885    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-openshift-service-ca\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.116871 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.116850    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-metrics-certs\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.124547 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.124518    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x96d\" (UniqueName: \"kubernetes.io/projected/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-kube-api-access-8x96d\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.619552 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.619510    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.622062 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.622033    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls\") pod \"maas-api-e2e-ait-c87701ad-5694955448-mp4mv\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.623862 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.623839    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:11.954147 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:11.954123    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"]
Nov 03 13:27:11.956016 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:27:11.955972    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7ba06d_1b55_4b6f_a1d3_500aca77c625.slice/crio-2a1d4066058549a6feeba955dd403f477242cff2e38262e235af10ab0c3dd507 WatchSource:0}: Error finding container 2a1d4066058549a6feeba955dd403f477242cff2e38262e235af10ab0c3dd507: Status 404 returned error can't find the container with id 2a1d4066058549a6feeba955dd403f477242cff2e38262e235af10ab0c3dd507
Nov 03 13:27:12.605525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:12.605489    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv" event={"ID":"4a7ba06d-1b55-4b6f-a1d3-500aca77c625","Type":"ContainerStarted","Data":"e4bf4f3d9d30eddabbd271dff29d5e98541b5c2d0721f2c702edad1fd523654b"}
Nov 03 13:27:12.605905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:12.605544    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv" event={"ID":"4a7ba06d-1b55-4b6f-a1d3-500aca77c625","Type":"ContainerStarted","Data":"2a1d4066058549a6feeba955dd403f477242cff2e38262e235af10ab0c3dd507"}
Nov 03 13:27:12.605905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:12.605565    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:12.627366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:12.627307    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv" podStartSLOduration=2.188835206 podStartE2EDuration="2.627292471s" totalImagesPullingTime="438.457265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:12.623028872 +0000 UTC m=+1120.434697080" watchObservedRunningTime="2026-11-03 13:27:12.627292471 +0000 UTC m=+1120.438960659"
Nov 03 13:27:13.222995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.222961    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"]
Nov 03 13:27:13.227400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.227377    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.242929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.242901    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.243056 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.242947    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.243056 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.243044    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlz75\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-kube-api-access-qlz75\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.243154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.243099    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.243214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.243172    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.243416 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.243387    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92849677-985f-4a8f-b812-bd24d20b7c8a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.243515 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.243451    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.243515 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.243484    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.243515 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.243511    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.259181 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.259150    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"]
Nov 03 13:27:13.344958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.344921    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92849677-985f-4a8f-b812-bd24d20b7c8a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.344965    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.344997    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345026    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345102    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345138    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345252    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlz75\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-kube-api-access-qlz75\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345320    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345376    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345455    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345533    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345627    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.345787 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.345762    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92849677-985f-4a8f-b812-bd24d20b7c8a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.346165 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.346145    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.347586 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.347567    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.347718 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.347701    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.354161 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.354141    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.354566 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.354545    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlz75\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-kube-api-access-qlz75\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.542591 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.542504    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:13.829697 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.829664    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"]
Nov 03 13:27:13.831457 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:27:13.831431    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92849677_985f_4a8f_b812_bd24d20b7c8a.slice/crio-bd2af8cd6e3f376140d261aa439bbe5fb11d3f6b140b26790ab715e23b39f916 WatchSource:0}: Error finding container bd2af8cd6e3f376140d261aa439bbe5fb11d3f6b140b26790ab715e23b39f916: Status 404 returned error can't find the container with id bd2af8cd6e3f376140d261aa439bbe5fb11d3f6b140b26790ab715e23b39f916
Nov 03 13:27:13.834361 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.834311    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:27:13.834462 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.834403    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:27:13.834462 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:13.834447    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:27:14.622403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:14.622367    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" event={"ID":"92849677-985f-4a8f-b812-bd24d20b7c8a","Type":"ContainerStarted","Data":"850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604"}
Nov 03 13:27:14.622600 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:14.622435    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" event={"ID":"92849677-985f-4a8f-b812-bd24d20b7c8a","Type":"ContainerStarted","Data":"bd2af8cd6e3f376140d261aa439bbe5fb11d3f6b140b26790ab715e23b39f916"}
Nov 03 13:27:14.641283 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:14.641221    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" podStartSLOduration=1.6412025300000002 podStartE2EDuration="1.64120253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:14.639167318 +0000 UTC m=+1122.450835510" watchObservedRunningTime="2026-11-03 13:27:14.64120253 +0000 UTC m=+1122.452870719"
Nov 03 13:27:15.559110 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:15.552734    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:15.560707 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:15.560676    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 13:27:15.560834 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:15.560742    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 13:27:16.546778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:16.546738    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 13:27:16.547035 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:16.547009    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 13:27:17.544911 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:17.544875    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 13:27:17.567635 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:17.567585    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 13:27:17.584407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:17.583531    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:17.586420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:17.585544    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:18.554366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:18.546738    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 13:27:18.554366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:18.546800    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 13:27:18.620733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:18.620702    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:19.547808 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:19.547767    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 13:27:19.548106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:19.548067    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 13:27:20.614305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:20.614267    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:20.614822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:20.614758    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:20.616889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:20.616859    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:27:32.575134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:32.575103    4447 scope.go:122] "RemoveContainer" containerID="361817f90c88b905e652ed90ea1ddfd04d6199c534225189172f3509af57bd0d"
Nov 03 13:27:33.411697 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:27:33.411660    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:27:35.171896 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:35.171861    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"]
Nov 03 13:27:35.198957 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:35.198921    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"]
Nov 03 13:27:36.457092 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.457051    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"]
Nov 03 13:27:36.463770 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.463744    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.482090 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.482056    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"]
Nov 03 13:27:36.491846 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.491773    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785077ae-674b-43c8-b7f5-34ee6278b212-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.492234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.491907    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.492234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.492004    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.492234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.492066    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.492234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.492093    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.492234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.492147    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.492234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.492170    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmcb\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-kube-api-access-ndmcb\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.492605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.492326    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.492605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.492401    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785077ae-674b-43c8-b7f5-34ee6278b212-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.511737 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.511706    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"]
Nov 03 13:27:36.523455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.523417    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.541593 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.540371    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"]
Nov 03 13:27:36.594074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594033    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.594074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594067    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887cf56-37a7-433e-9baf-27e0ded56402-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.594333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594106    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785077ae-674b-43c8-b7f5-34ee6278b212-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.594333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594146    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.594333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594195    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785077ae-674b-43c8-b7f5-34ee6278b212-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.594333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594253    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.594636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594455    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.594636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594493    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.594636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594532    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.594636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594576    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887cf56-37a7-433e-9baf-27e0ded56402-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.594858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594663    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.594858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594694    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.594858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594723    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.594858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594763    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.594858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594811    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.594858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594839    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmcb\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-kube-api-access-ndmcb\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.595163 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594867    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.595163 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594952    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwr5w\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-kube-api-access-zwr5w\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.595163 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.594999    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.595163 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.595013    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785077ae-674b-43c8-b7f5-34ee6278b212-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.595381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.595315    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.595381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.595317    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.596403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.596376    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.596908 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.596864    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.597312 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.597293    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785077ae-674b-43c8-b7f5-34ee6278b212-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.605284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.605255    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmcb\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-kube-api-access-ndmcb\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.605506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.605487    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.696715 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.696675    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887cf56-37a7-433e-9baf-27e0ded56402-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.696882 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.696763    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.696882 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.696790    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.696882 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.696833    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwr5w\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-kube-api-access-zwr5w\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697042 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.696899    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887cf56-37a7-433e-9baf-27e0ded56402-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697042 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.696946    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697042 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.697003    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.697051    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.697085    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697304 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.697214    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697304 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.697246    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.697439    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.697521    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.697539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.697526    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887cf56-37a7-433e-9baf-27e0ded56402-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.699650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.699616    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887cf56-37a7-433e-9baf-27e0ded56402-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.700082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.700055    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.704810 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.704777    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.704929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.704822    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwr5w\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-kube-api-access-zwr5w\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:36.787892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.787813    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:36.850700 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:36.850674    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:37.104295 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.104263    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"]
Nov 03 13:27:37.108411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.108371    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:27:37.108510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.108459    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:27:37.108510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.108499    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:27:37.158465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.158421    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"]
Nov 03 13:27:37.162443 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:27:37.162416    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5887cf56_37a7_433e_9baf_27e0ded56402.slice/crio-ade7825e367ffae1af4f03c67b3538588be403fe227f277f85ff56106db4e673 WatchSource:0}: Error finding container ade7825e367ffae1af4f03c67b3538588be403fe227f277f85ff56106db4e673: Status 404 returned error can't find the container with id ade7825e367ffae1af4f03c67b3538588be403fe227f277f85ff56106db4e673
Nov 03 13:27:37.166754 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.166554    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:27:37.166754 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.166635    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:27:37.167743 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.167231    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:27:37.896863 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.895151    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" event={"ID":"5887cf56-37a7-433e-9baf-27e0ded56402","Type":"ContainerStarted","Data":"586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c"}
Nov 03 13:27:37.896863 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.895260    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" event={"ID":"5887cf56-37a7-433e-9baf-27e0ded56402","Type":"ContainerStarted","Data":"ade7825e367ffae1af4f03c67b3538588be403fe227f277f85ff56106db4e673"}
Nov 03 13:27:37.904271 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.904239    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" event={"ID":"785077ae-674b-43c8-b7f5-34ee6278b212","Type":"ContainerStarted","Data":"8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9"}
Nov 03 13:27:37.904548 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.904526    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" event={"ID":"785077ae-674b-43c8-b7f5-34ee6278b212","Type":"ContainerStarted","Data":"1e6992663dfa14c029405f4dc4160c9b21161632e75830ea99d830b5128100b5"}
Nov 03 13:27:37.961367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:37.935896    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podStartSLOduration=1.935875272 podStartE2EDuration="1.935875272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:37.922627531 +0000 UTC m=+1145.734295750" watchObservedRunningTime="2026-11-03 13:27:37.935875272 +0000 UTC m=+1145.747543462"
Nov 03 13:27:38.020367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:38.013278    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" podStartSLOduration=2.013259274 podStartE2EDuration="2.013259274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:38.002818771 +0000 UTC m=+1145.814486986" watchObservedRunningTime="2026-11-03 13:27:38.013259274 +0000 UTC m=+1145.824927463"
Nov 03 13:27:38.792174 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:38.792138    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:38.807287 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:38.807249    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:27:38.807580 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:38.807548    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 13:27:38.855311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:38.855231    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:38.855772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:38.855747    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 13:27:38.855948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:38.855923    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 13:27:39.800267 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:39.800234    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:27:39.801353 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:39.801300    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 13:27:39.855900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:39.855863    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 13:27:39.856179 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:39.856151    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 13:27:40.174517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:40.174417    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz" podUID="4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63" containerName="payload-processing" containerID="cri-o://62e618f78efdf3f856eed93448c73fa7c53895c946c25d46b17af683d329c216" gracePeriod=25
Nov 03 13:27:40.227879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:40.214493    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq" podUID="79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f" containerName="payload-pre-processing" containerID="cri-o://7b74d061973675d2b994432a2f0f86dafe73f20bc85e465d70a00dae92c7939c" gracePeriod=25
Nov 03 13:27:40.227879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:40.215297    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"]
Nov 03 13:27:40.227879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:40.215549    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv" podUID="4a7ba06d-1b55-4b6f-a1d3-500aca77c625" containerName="maas-api" containerID="cri-o://e4bf4f3d9d30eddabbd271dff29d5e98541b5c2d0721f2c702edad1fd523654b" gracePeriod=30
Nov 03 13:27:40.979415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:40.952847    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 13:27:40.979415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:40.952940    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 13:27:40.979415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:40.953403    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:27:40.979415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:40.953446    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 13:27:41.436465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:41.430335    4447 generic.go:356] "Generic (PLEG): container finished" podID="79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f" containerID="7b74d061973675d2b994432a2f0f86dafe73f20bc85e465d70a00dae92c7939c" exitCode=0
Nov 03 13:27:41.436465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:41.430504    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq" event={"ID":"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f","Type":"ContainerDied","Data":"7b74d061973675d2b994432a2f0f86dafe73f20bc85e465d70a00dae92c7939c"}
Nov 03 13:27:41.512267 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:41.512216    4447 generic.go:356] "Generic (PLEG): container finished" podID="4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63" containerID="62e618f78efdf3f856eed93448c73fa7c53895c946c25d46b17af683d329c216" exitCode=0
Nov 03 13:27:41.512816 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:41.512789    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz" event={"ID":"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63","Type":"ContainerDied","Data":"62e618f78efdf3f856eed93448c73fa7c53895c946c25d46b17af683d329c216"}
Nov 03 13:27:41.541289 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:41.533968    4447 generic.go:356] "Generic (PLEG): container finished" podID="4a7ba06d-1b55-4b6f-a1d3-500aca77c625" containerID="e4bf4f3d9d30eddabbd271dff29d5e98541b5c2d0721f2c702edad1fd523654b" exitCode=0
Nov 03 13:27:41.541289 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:41.534070    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv" event={"ID":"4a7ba06d-1b55-4b6f-a1d3-500aca77c625","Type":"ContainerDied","Data":"e4bf4f3d9d30eddabbd271dff29d5e98541b5c2d0721f2c702edad1fd523654b"}
Nov 03 13:27:41.807728 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:41.789081    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:27:41.807728 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:41.789142    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 13:27:42.210799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.197488    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 13:27:42.210799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.197558    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 13:27:42.339612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.325293    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:42.549369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.503882    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls\") pod \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") "
Nov 03 13:27:42.549369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.503991    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-kube-api-access-8x96d\" (UniqueName: \"kubernetes.io/projected/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-kube-api-access-8x96d\") pod \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") "
Nov 03 13:27:42.549369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.504067    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-openshift-service-ca\") pod \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") "
Nov 03 13:27:42.549369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.504166    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-metrics-certs\") pod \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\" (UID: \"4a7ba06d-1b55-4b6f-a1d3-500aca77c625\") "
Nov 03 13:27:42.582779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.550204    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-metrics-certs" pod "4a7ba06d-1b55-4b6f-a1d3-500aca77c625" (UID: "4a7ba06d-1b55-4b6f-a1d3-500aca77c625"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:27:42.662457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.660106    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-kube-api-access-8x96d" pod "4a7ba06d-1b55-4b6f-a1d3-500aca77c625" (UID: "4a7ba06d-1b55-4b6f-a1d3-500aca77c625"). InnerVolumeSpecName "kube-api-access-8x96d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:42.662457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.660232    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls" pod "4a7ba06d-1b55-4b6f-a1d3-500aca77c625" (UID: "4a7ba06d-1b55-4b6f-a1d3-500aca77c625"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:27:42.662704 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.662516    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-openshift-service-ca" pod "4a7ba06d-1b55-4b6f-a1d3-500aca77c625" (UID: "4a7ba06d-1b55-4b6f-a1d3-500aca77c625"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:42.693146 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.693108    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x96d\" (UniqueName: \"kubernetes.io/projected/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-kube-api-access-8x96d\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:42.693420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.693384    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:42.693562 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.693547    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:42.693656 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.693644    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a7ba06d-1b55-4b6f-a1d3-500aca77c625-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:42.805074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.804980    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:27:42.805663 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.805628    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 13:27:42.814361 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.814321    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv" event={"ID":"4a7ba06d-1b55-4b6f-a1d3-500aca77c625","Type":"ContainerDied","Data":"2a1d4066058549a6feeba955dd403f477242cff2e38262e235af10ab0c3dd507"}
Nov 03 13:27:42.814543 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.814530    4447 scope.go:122] "RemoveContainer" containerID="e4bf4f3d9d30eddabbd271dff29d5e98541b5c2d0721f2c702edad1fd523654b"
Nov 03 13:27:42.814747 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.814737    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"
Nov 03 13:27:42.827239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.827215    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:42.912005 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.911974    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-kube-api-access-krbvv\" (UniqueName: \"kubernetes.io/projected/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-kube-api-access-krbvv\") pod \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\" (UID: \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\") "
Nov 03 13:27:42.912158 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.912084    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-plugins-config-volume\") pod \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\" (UID: \"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f\") "
Nov 03 13:27:42.920363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.913123    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-plugins-config-volume" pod "79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f" (UID: "79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:42.920363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.915231    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"]
Nov 03 13:27:42.922398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.922370    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c87701ad-5694955448-mp4mv"]
Nov 03 13:27:42.924074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.924048    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 13:27:42.924503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.924478    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 13:27:42.932560 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:42.932536    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-kube-api-access-krbvv" pod "79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f" (UID: "79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f"). InnerVolumeSpecName "kube-api-access-krbvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:43.020482 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.020441    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:43.022370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.022306    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbvv\" (UniqueName: \"kubernetes.io/projected/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f-kube-api-access-krbvv\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:43.161367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.135820    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:43.504473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.498405    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-kube-api-access-flrfd\" (UniqueName: \"kubernetes.io/projected/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-kube-api-access-flrfd\") pod \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\" (UID: \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\") "
Nov 03 13:27:43.511409 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.506314    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-plugins-config-volume\") pod \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\" (UID: \"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63\") "
Nov 03 13:27:43.522563 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.515593    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-plugins-config-volume" pod "4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63" (UID: "4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:43.542979 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.542922    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-kube-api-access-flrfd" pod "4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63" (UID: "4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63"). InnerVolumeSpecName "kube-api-access-flrfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:43.626555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.626523    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:43.626753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.626613    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flrfd\" (UniqueName: \"kubernetes.io/projected/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63-kube-api-access-flrfd\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:43.791486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.790807    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:27:43.791486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.790895    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 13:27:43.869369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.866132    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 13:27:43.869369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.866190    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 13:27:43.917367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.907708    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq" event={"ID":"79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f","Type":"ContainerDied","Data":"26be07ba6afd7942cb609af0c750a788d19827b505b64128c0e483f13996f85b"}
Nov 03 13:27:43.917367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.907783    4447 scope.go:122] "RemoveContainer" containerID="7b74d061973675d2b994432a2f0f86dafe73f20bc85e465d70a00dae92c7939c"
Nov 03 13:27:43.917367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.907918    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"
Nov 03 13:27:43.918475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.918085    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz" event={"ID":"4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63","Type":"ContainerDied","Data":"7af098cda58e1bb40f29eaa1152cfc40fc0656f5a364278a6cd8d9655d67051e"}
Nov 03 13:27:43.918475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.918197    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"
Nov 03 13:27:43.933946 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.933927    4447 scope.go:122] "RemoveContainer" containerID="62e618f78efdf3f856eed93448c73fa7c53895c946c25d46b17af683d329c216"
Nov 03 13:27:43.974365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:43.968410    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"]
Nov 03 13:27:44.025545 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.025420    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c87701ad-b469d86dc-4mcbq"]
Nov 03 13:27:44.029733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.029706    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"]
Nov 03 13:27:44.033938 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.033914    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c87701ad-6ccb48d5c7-hzscz"]
Nov 03 13:27:44.321393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.321366    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7ba06d-1b55-4b6f-a1d3-500aca77c625" path="/var/lib/kubelet/pods/4a7ba06d-1b55-4b6f-a1d3-500aca77c625/volumes"
Nov 03 13:27:44.322081 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.322054    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63" path="/var/lib/kubelet/pods/4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63/volumes"
Nov 03 13:27:44.322487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.322472    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f" path="/var/lib/kubelet/pods/79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f/volumes"
Nov 03 13:27:44.789036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.788996    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:27:44.789630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.789071    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 13:27:44.855367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.854730    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 13:27:44.855367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:44.854792    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 13:27:45.851766 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:45.851731    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 13:27:45.852158 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:45.851802    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 13:27:46.242066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:46.242033    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:46.242769 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:46.242745    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:46.243754 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:46.243732    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:27:46.977549 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:46.977517    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:46.978257 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:46.978190    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:46.979642 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:46.979591    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:27:51.042120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042087    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"]
Nov 03 13:27:51.042730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042716    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63" containerName="payload-processing"
Nov 03 13:27:51.042774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042732    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63" containerName="payload-processing"
Nov 03 13:27:51.042774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042751    4447 cpu_manager.go:414] "containerMap: removing container" podUID="79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f" containerName="payload-pre-processing"
Nov 03 13:27:51.042774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042759    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f" containerName="payload-pre-processing"
Nov 03 13:27:51.042863 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042788    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4a7ba06d-1b55-4b6f-a1d3-500aca77c625" containerName="maas-api"
Nov 03 13:27:51.042863 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042794    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7ba06d-1b55-4b6f-a1d3-500aca77c625" containerName="maas-api"
Nov 03 13:27:51.042924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042898    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cda5853-4fc9-41ce-8eaa-78cfa1cd2e63" containerName="payload-processing"
Nov 03 13:27:51.042924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042910    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bb861f-b035-4ffa-9ce3-2eb5be3c5e4f" containerName="payload-pre-processing"
Nov 03 13:27:51.042924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.042918    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7ba06d-1b55-4b6f-a1d3-500aca77c625" containerName="maas-api"
Nov 03 13:27:51.047959 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.047937    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.056590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.056559    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a62d22-serving-cert\""
Nov 03 13:27:51.061120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.061093    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"]
Nov 03 13:27:51.108239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.108202    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.108450 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.108373    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-metrics-certs\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.108511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.108450    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72l9p\" (UniqueName: \"kubernetes.io/projected/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-kube-api-access-72l9p\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.108511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.108502    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.210018 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.209986    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.210189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.210092    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.210245 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.210229    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-metrics-certs\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.210326 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.210311    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72l9p\" (UniqueName: \"kubernetes.io/projected/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-kube-api-access-72l9p\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.210854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.210827    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.212651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.212632    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-metrics-certs\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.212724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.212672    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.219022 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.218995    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72l9p\" (UniqueName: \"kubernetes.io/projected/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-kube-api-access-72l9p\") pod \"maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.364415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.363795    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:51.669380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.669254    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"]
Nov 03 13:27:51.673923 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:27:51.673892    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9bbb1b_dadb_4d79_a999_dc4f32fcb64e.slice/crio-762c579e61b267d184f0292ccf4fd622ca18cf946a4d052938cd96c81706212b WatchSource:0}: Error finding container 762c579e61b267d184f0292ccf4fd622ca18cf946a4d052938cd96c81706212b: Status 404 returned error can't find the container with id 762c579e61b267d184f0292ccf4fd622ca18cf946a4d052938cd96c81706212b
Nov 03 13:27:51.993286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:51.993244    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv" event={"ID":"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e","Type":"ContainerStarted","Data":"762c579e61b267d184f0292ccf4fd622ca18cf946a4d052938cd96c81706212b"}
Nov 03 13:27:53.001606 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.001567    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv" event={"ID":"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e","Type":"ContainerStarted","Data":"c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa"}
Nov 03 13:27:53.002055 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.001727    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:27:53.033808 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.033742    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv" podStartSLOduration=1.6232329380000001 podStartE2EDuration="2.033721609s" totalImagesPullingTime="410.488671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:53.030846058 +0000 UTC m=+1160.842514279" watchObservedRunningTime="2026-11-03 13:27:53.033721609 +0000 UTC m=+1160.845389801"
Nov 03 13:27:53.168751 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.168717    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"]
Nov 03 13:27:53.173178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.173158    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:27:53.182415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.182393    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ef15d972-dockercfg-wfw52\""
Nov 03 13:27:53.187039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.187021    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ef15d972\""
Nov 03 13:27:53.219766 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.219729    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"]
Nov 03 13:27:53.255641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.254200    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl69b\" (UniqueName: \"kubernetes.io/projected/38c202cb-89eb-492e-a328-656477c0c60a-kube-api-access-xl69b\") pod \"payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f\" (UID: \"38c202cb-89eb-492e-a328-656477c0c60a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:27:53.255641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.254363    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c202cb-89eb-492e-a328-656477c0c60a-plugins-config-volume\") pod \"payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f\" (UID: \"38c202cb-89eb-492e-a328-656477c0c60a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:27:53.356184 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.356150    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl69b\" (UniqueName: \"kubernetes.io/projected/38c202cb-89eb-492e-a328-656477c0c60a-kube-api-access-xl69b\") pod \"payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f\" (UID: \"38c202cb-89eb-492e-a328-656477c0c60a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:27:53.356336 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.356308    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c202cb-89eb-492e-a328-656477c0c60a-plugins-config-volume\") pod \"payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f\" (UID: \"38c202cb-89eb-492e-a328-656477c0c60a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:27:53.357017 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.356991    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c202cb-89eb-492e-a328-656477c0c60a-plugins-config-volume\") pod \"payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f\" (UID: \"38c202cb-89eb-492e-a328-656477c0c60a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:27:53.371863 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.371834    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl69b\" (UniqueName: \"kubernetes.io/projected/38c202cb-89eb-492e-a328-656477c0c60a-kube-api-access-xl69b\") pod \"payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f\" (UID: \"38c202cb-89eb-492e-a328-656477c0c60a\") " pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:27:53.489016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.488982    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:27:53.797431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:53.797401    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"]
Nov 03 13:27:53.800723 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:27:53.800690    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c202cb_89eb_492e_a328_656477c0c60a.slice/crio-0e34e68918806be4b000c76563faea366582e7feb66d18c99ddc690977ca1b07 WatchSource:0}: Error finding container 0e34e68918806be4b000c76563faea366582e7feb66d18c99ddc690977ca1b07: Status 404 returned error can't find the container with id 0e34e68918806be4b000c76563faea366582e7feb66d18c99ddc690977ca1b07
Nov 03 13:27:54.011387 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:54.011326    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f" event={"ID":"38c202cb-89eb-492e-a328-656477c0c60a","Type":"ContainerStarted","Data":"b498fc80ad6275a366636ec2752ed4d8361cd6a2cd61cc03e08451e715499d2e"}
Nov 03 13:27:54.011896 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:54.011401    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f" event={"ID":"38c202cb-89eb-492e-a328-656477c0c60a","Type":"ContainerStarted","Data":"0e34e68918806be4b000c76563faea366582e7feb66d18c99ddc690977ca1b07"}
Nov 03 13:27:54.011896 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:54.011602    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:27:54.033514 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:54.033464    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f" podStartSLOduration=1.033451993 podStartE2EDuration="1.033451993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:54.032529211 +0000 UTC m=+1161.844197441" watchObservedRunningTime="2026-11-03 13:27:54.033451993 +0000 UTC m=+1161.845120179"
Nov 03 13:27:59.027576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:27:59.027536    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:28:00.938696 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:00.938659    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"]
Nov 03 13:28:00.944378 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:00.944319    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:00.959547 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:00.959521    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ef15d972-serving-cert\""
Nov 03 13:28:00.972149 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:00.972119    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"]
Nov 03 13:28:01.046000 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.045964    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-maas-api-tls\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.046151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.046095    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6q8\" (UniqueName: \"kubernetes.io/projected/82e9e3c5-0977-4a89-a149-12f93e791504-kube-api-access-9f6q8\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.046205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.046186    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-metrics-certs\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.046336 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.046312    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e9e3c5-0977-4a89-a149-12f93e791504-openshift-service-ca\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.147744 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.147683    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-maas-api-tls\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.147961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.147800    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6q8\" (UniqueName: \"kubernetes.io/projected/82e9e3c5-0977-4a89-a149-12f93e791504-kube-api-access-9f6q8\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.147961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.147877    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-metrics-certs\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.147961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.147939    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e9e3c5-0977-4a89-a149-12f93e791504-openshift-service-ca\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.148711 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.148684    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e9e3c5-0977-4a89-a149-12f93e791504-openshift-service-ca\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.151080 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.151058    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-maas-api-tls\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.151712 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.151614    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-metrics-certs\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.157975 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.157946    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6q8\" (UniqueName: \"kubernetes.io/projected/82e9e3c5-0977-4a89-a149-12f93e791504-kube-api-access-9f6q8\") pod \"maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.270021 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.269933    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:01.571446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:01.562788    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"]
Nov 03 13:28:01.571446 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:28:01.563877    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e9e3c5_0977_4a89_a149_12f93e791504.slice/crio-ed89777147ceb68d8b924ad158f0bc81a64a693df331f68f1bdf4a0e0b2b1c0d WatchSource:0}: Error finding container ed89777147ceb68d8b924ad158f0bc81a64a693df331f68f1bdf4a0e0b2b1c0d: Status 404 returned error can't find the container with id ed89777147ceb68d8b924ad158f0bc81a64a693df331f68f1bdf4a0e0b2b1c0d
Nov 03 13:28:02.153912 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:02.153866    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6" event={"ID":"82e9e3c5-0977-4a89-a149-12f93e791504","Type":"ContainerStarted","Data":"381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c"}
Nov 03 13:28:02.154332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:02.153942    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6" event={"ID":"82e9e3c5-0977-4a89-a149-12f93e791504","Type":"ContainerStarted","Data":"ed89777147ceb68d8b924ad158f0bc81a64a693df331f68f1bdf4a0e0b2b1c0d"}
Nov 03 13:28:02.154332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:02.154087    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:02.192193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:02.192141    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6" podStartSLOduration=1.742313921 podStartE2EDuration="2.192127546s" totalImagesPullingTime="449.813625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:02.19180866 +0000 UTC m=+1170.003476873" watchObservedRunningTime="2026-11-03 13:28:02.192127546 +0000 UTC m=+1170.003795782"
Nov 03 13:28:03.415186 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:03.415055    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:28:05.023426 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:05.023398    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:28:08.167198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:08.167167    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:14.745048 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.745011    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"]
Nov 03 13:28:14.749865 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.749835    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.759429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.759400    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-29dd648f7d07\""
Nov 03 13:28:14.759607 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.759589    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-f5ffe280c5d9\""
Nov 03 13:28:14.759709 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.759675    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-e2e--cd1da41a-dockercfg-txc88\""
Nov 03 13:28:14.762377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.762335    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c192b\""
Nov 03 13:28:14.762507 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.762411    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"openshift-service-ca.crt\""
Nov 03 13:28:14.762570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.762505    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"routing-overlay\""
Nov 03 13:28:14.762630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.762571    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"kube-root-ca.crt\""
Nov 03 13:28:14.775498 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.775472    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"]
Nov 03 13:28:14.852048 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.852000    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxvr\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-kube-api-access-vpxvr\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.852429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.852406    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.853510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.852582    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.853510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.852624    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.967189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.956478    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.967189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.956681    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.967189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.956747    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.967189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.956859    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxvr\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-kube-api-access-vpxvr\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.967189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.958047    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.981660 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.979404    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.995578 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.995486    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxvr\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-kube-api-access-vpxvr\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:14.997774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:14.997742    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:15.075364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:15.074619    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:15.430453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:15.430422    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"]
Nov 03 13:28:15.430720 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:28:15.430697    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9728df0b_e653_4464_9f46_771b8a107389.slice/crio-4abcebe790719f6fd7779b743014513d913f518f9f6acb37df580e4eb30a742a WatchSource:0}: Error finding container 4abcebe790719f6fd7779b743014513d913f518f9f6acb37df580e4eb30a742a: Status 404 returned error can't find the container with id 4abcebe790719f6fd7779b743014513d913f518f9f6acb37df580e4eb30a742a
Nov 03 13:28:16.308276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:16.308240    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" event={"ID":"9728df0b-e653-4464-9f46-771b8a107389","Type":"ContainerStarted","Data":"409f7accc51a814f4e8d17d165d4bdebb908f4f6c5077818f88ceb68c8a9b1b3"}
Nov 03 13:28:16.308753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:16.308394    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" event={"ID":"9728df0b-e653-4464-9f46-771b8a107389","Type":"ContainerStarted","Data":"4abcebe790719f6fd7779b743014513d913f518f9f6acb37df580e4eb30a742a"}
Nov 03 13:28:16.308753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:16.308437    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:16.343438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:16.343384    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" podStartSLOduration=2.343371451 podStartE2EDuration="2.343371451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:16.343055155 +0000 UTC m=+1184.154723364" watchObservedRunningTime="2026-11-03 13:28:16.343371451 +0000 UTC m=+1184.155039638"
Nov 03 13:28:24.833445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:24.833409    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"]
Nov 03 13:28:24.838218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:24.838190    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:24.851100 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:24.851075    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-cf53a4-serving-cert\""
Nov 03 13:28:24.853298 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:24.853273    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"]
Nov 03 13:28:24.933737 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:24.933697    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bc340b-451f-4c34-96ad-b514783d1f88-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:24.933930 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:24.933753    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-metrics-certs\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:24.934000 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:24.933953    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkbz\" (UniqueName: \"kubernetes.io/projected/73bc340b-451f-4c34-96ad-b514783d1f88-kube-api-access-dbkbz\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:24.934000 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:24.933985    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.035530 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.035499    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bc340b-451f-4c34-96ad-b514783d1f88-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.035706 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.035536    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-metrics-certs\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.035706 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.035632    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkbz\" (UniqueName: \"kubernetes.io/projected/73bc340b-451f-4c34-96ad-b514783d1f88-kube-api-access-dbkbz\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.035706 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.035652    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.036300 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.036271    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bc340b-451f-4c34-96ad-b514783d1f88-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.038190 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.038161    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-metrics-certs\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.038306 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.038169    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.044623 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.044595    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkbz\" (UniqueName: \"kubernetes.io/projected/73bc340b-451f-4c34-96ad-b514783d1f88-kube-api-access-dbkbz\") pod \"maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.155838 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.155761    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:25.458427 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:25.458371    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"]
Nov 03 13:28:25.460054 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:28:25.460022    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bc340b_451f_4c34_96ad_b514783d1f88.slice/crio-a1ac0560c1fc19d1ee6c29ee53a30b1e9ab2f5811fe4b965f9b949dae04324f9 WatchSource:0}: Error finding container a1ac0560c1fc19d1ee6c29ee53a30b1e9ab2f5811fe4b965f9b949dae04324f9: Status 404 returned error can't find the container with id a1ac0560c1fc19d1ee6c29ee53a30b1e9ab2f5811fe4b965f9b949dae04324f9
Nov 03 13:28:26.397746 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:26.397711    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766" event={"ID":"73bc340b-451f-4c34-96ad-b514783d1f88","Type":"ContainerStarted","Data":"9e7ec687b647637969ae5f34088514188ebea8a9ca12be39c2efbbbe1aae765c"}
Nov 03 13:28:26.398223 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:26.397762    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766" event={"ID":"73bc340b-451f-4c34-96ad-b514783d1f88","Type":"ContainerStarted","Data":"a1ac0560c1fc19d1ee6c29ee53a30b1e9ab2f5811fe4b965f9b949dae04324f9"}
Nov 03 13:28:26.398223 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:26.397833    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:26.444936 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:26.443943    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766" podStartSLOduration=1.983397745 podStartE2EDuration="2.443923649s" totalImagesPullingTime="460.525904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:26.441859036 +0000 UTC m=+1194.253527225" watchObservedRunningTime="2026-11-03 13:28:26.443923649 +0000 UTC m=+1194.255591855"
Nov 03 13:28:27.318483 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:27.318455    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:27.379182 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:27.378832    4447 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7c192b/agc-cred-f5ffe280c5d9: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:27.379182 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:27.378861    4447 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:27.379182 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:27.378943    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials podName:9728df0b-e653-4464-9f46-771b8a107389 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:27.878923026 +0000 UTC m=+1195.690591210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" (UID: "9728df0b-e653-4464-9f46-771b8a107389") : secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:27.889961 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:27.889930    4447 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7c192b/agc-cred-f5ffe280c5d9: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:27.889961 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:27.889956    4447 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:27.890426 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:27.890008    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials podName:9728df0b-e653-4464-9f46-771b8a107389 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:28.889994058 +0000 UTC m=+1196.701662237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" (UID: "9728df0b-e653-4464-9f46-771b8a107389") : secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:28.262155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:28.262118    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"]
Nov 03 13:28:28.907049 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:28.907010    4447 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7c192b/agc-cred-f5ffe280c5d9: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:28.907551 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:28.907065    4447 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:28.907551 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:28.907133    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials podName:9728df0b-e653-4464-9f46-771b8a107389 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:30.907116092 +0000 UTC m=+1198.718784281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" (UID: "9728df0b-e653-4464-9f46-771b8a107389") : secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:30.938986 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:30.938946    4447 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7c192b/agc-cred-f5ffe280c5d9: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:30.938986 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:30.938978    4447 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:30.939568 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:30.939071    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials podName:9728df0b-e653-4464-9f46-771b8a107389 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:34.939053781 +0000 UTC m=+1202.750721964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" (UID: "9728df0b-e653-4464-9f46-771b8a107389") : secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:32.412279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:32.412250    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:28:33.262673 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.262635    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f" podUID="38c202cb-89eb-492e-a328-656477c0c60a" containerName="payload-processing" containerID="cri-o://b498fc80ad6275a366636ec2752ed4d8361cd6a2cd61cc03e08451e715499d2e" gracePeriod=25
Nov 03 13:28:33.411489 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:33.411448    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:28:33.470792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.470675    4447 generic.go:356] "Generic (PLEG): container finished" podID="38c202cb-89eb-492e-a328-656477c0c60a" containerID="b498fc80ad6275a366636ec2752ed4d8361cd6a2cd61cc03e08451e715499d2e" exitCode=0
Nov 03 13:28:33.470792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.470737    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f" event={"ID":"38c202cb-89eb-492e-a328-656477c0c60a","Type":"ContainerDied","Data":"b498fc80ad6275a366636ec2752ed4d8361cd6a2cd61cc03e08451e715499d2e"}
Nov 03 13:28:33.554694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.554672    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:28:33.677891 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.677859    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c202cb-89eb-492e-a328-656477c0c60a-kube-api-access-xl69b\" (UniqueName: \"kubernetes.io/projected/38c202cb-89eb-492e-a328-656477c0c60a-kube-api-access-xl69b\") pod \"38c202cb-89eb-492e-a328-656477c0c60a\" (UID: \"38c202cb-89eb-492e-a328-656477c0c60a\") "
Nov 03 13:28:33.678123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.678058    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c202cb-89eb-492e-a328-656477c0c60a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c202cb-89eb-492e-a328-656477c0c60a-plugins-config-volume\") pod \"38c202cb-89eb-492e-a328-656477c0c60a\" (UID: \"38c202cb-89eb-492e-a328-656477c0c60a\") "
Nov 03 13:28:33.679165 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.679123    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c202cb-89eb-492e-a328-656477c0c60a-plugins-config-volume" pod "38c202cb-89eb-492e-a328-656477c0c60a" (UID: "38c202cb-89eb-492e-a328-656477c0c60a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:33.694823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.693680    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"]
Nov 03 13:28:33.694823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.693909    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6" podUID="82e9e3c5-0977-4a89-a149-12f93e791504" containerName="maas-api" containerID="cri-o://381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c" gracePeriod=30
Nov 03 13:28:33.696735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.696688    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c202cb-89eb-492e-a328-656477c0c60a-kube-api-access-xl69b" pod "38c202cb-89eb-492e-a328-656477c0c60a" (UID: "38c202cb-89eb-492e-a328-656477c0c60a"). InnerVolumeSpecName "kube-api-access-xl69b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:33.698129 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.698103    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl69b\" (UniqueName: \"kubernetes.io/projected/38c202cb-89eb-492e-a328-656477c0c60a-kube-api-access-xl69b\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:33.698244 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:33.698132    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c202cb-89eb-492e-a328-656477c0c60a-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:34.079106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.078880    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:34.129775 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.129737    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e9e3c5-0977-4a89-a149-12f93e791504-kube-api-access-9f6q8\" (UniqueName: \"kubernetes.io/projected/82e9e3c5-0977-4a89-a149-12f93e791504-kube-api-access-9f6q8\") pod \"82e9e3c5-0977-4a89-a149-12f93e791504\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") "
Nov 03 13:28:34.130416 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.130393    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e9e3c5-0977-4a89-a149-12f93e791504-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e9e3c5-0977-4a89-a149-12f93e791504-openshift-service-ca\") pod \"82e9e3c5-0977-4a89-a149-12f93e791504\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") "
Nov 03 13:28:34.137691 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.137657    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-metrics-certs\") pod \"82e9e3c5-0977-4a89-a149-12f93e791504\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") "
Nov 03 13:28:34.137829 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.137813    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-maas-api-tls\") pod \"82e9e3c5-0977-4a89-a149-12f93e791504\" (UID: \"82e9e3c5-0977-4a89-a149-12f93e791504\") "
Nov 03 13:28:34.142128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.142084    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-maas-api-tls" pod "82e9e3c5-0977-4a89-a149-12f93e791504" (UID: "82e9e3c5-0977-4a89-a149-12f93e791504"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:28:34.143754 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.143725    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-metrics-certs" pod "82e9e3c5-0977-4a89-a149-12f93e791504" (UID: "82e9e3c5-0977-4a89-a149-12f93e791504"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:28:34.145949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.145922    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e9e3c5-0977-4a89-a149-12f93e791504-openshift-service-ca" pod "82e9e3c5-0977-4a89-a149-12f93e791504" (UID: "82e9e3c5-0977-4a89-a149-12f93e791504"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:34.146061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.145927    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e9e3c5-0977-4a89-a149-12f93e791504-kube-api-access-9f6q8" pod "82e9e3c5-0977-4a89-a149-12f93e791504" (UID: "82e9e3c5-0977-4a89-a149-12f93e791504"). InnerVolumeSpecName "kube-api-access-9f6q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:34.240101 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.240062    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:34.240101 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.240091    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6q8\" (UniqueName: \"kubernetes.io/projected/82e9e3c5-0977-4a89-a149-12f93e791504-kube-api-access-9f6q8\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:34.240101 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.240102    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e9e3c5-0977-4a89-a149-12f93e791504-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:34.240337 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.240111    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e9e3c5-0977-4a89-a149-12f93e791504-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:34.479185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.479143    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"
Nov 03 13:28:34.479185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.479166    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f" event={"ID":"38c202cb-89eb-492e-a328-656477c0c60a","Type":"ContainerDied","Data":"0e34e68918806be4b000c76563faea366582e7feb66d18c99ddc690977ca1b07"}
Nov 03 13:28:34.479844 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.479237    4447 scope.go:122] "RemoveContainer" containerID="b498fc80ad6275a366636ec2752ed4d8361cd6a2cd61cc03e08451e715499d2e"
Nov 03 13:28:34.481534 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.481500    4447 generic.go:356] "Generic (PLEG): container finished" podID="82e9e3c5-0977-4a89-a149-12f93e791504" containerID="381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c" exitCode=0
Nov 03 13:28:34.481706 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.481597    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6" event={"ID":"82e9e3c5-0977-4a89-a149-12f93e791504","Type":"ContainerDied","Data":"381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c"}
Nov 03 13:28:34.481706 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.481643    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6" event={"ID":"82e9e3c5-0977-4a89-a149-12f93e791504","Type":"ContainerDied","Data":"ed89777147ceb68d8b924ad158f0bc81a64a693df331f68f1bdf4a0e0b2b1c0d"}
Nov 03 13:28:34.481905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.481713    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"
Nov 03 13:28:34.493097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.493061    4447 scope.go:122] "RemoveContainer" containerID="381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c"
Nov 03 13:28:34.503474 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.502702    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"]
Nov 03 13:28:34.509580 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.509553    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef15d972-6c89559d89-rrz7f"]
Nov 03 13:28:34.511106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.511070    4447 scope.go:122] "RemoveContainer" containerID="381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c"
Nov 03 13:28:34.511475 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:34.511454    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c\": container with ID starting with 381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c not found: ID does not exist" containerID="381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c"
Nov 03 13:28:34.511539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.511483    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c"} err="failed to get container status \"381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c\": rpc error: code = NotFound desc = could not find container \"381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c\": container with ID starting with 381f339f53e475a17c1a38804532a10192557cda78c6530be8874ab743dd8f9c not found: ID does not exist"
Nov 03 13:28:34.520331 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.520305    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"]
Nov 03 13:28:34.525097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:34.525063    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef15d972-58956c8cc7-69ls6"]
Nov 03 13:28:34.953237 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:34.953203    4447 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7c192b/agc-cred-f5ffe280c5d9: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:34.953237 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:34.953229    4447 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:34.953473 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:34.953281    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials podName:9728df0b-e653-4464-9f46-771b8a107389 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:42.953266627 +0000 UTC m=+1210.764934810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" (UID: "9728df0b-e653-4464-9f46-771b8a107389") : secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:36.320020 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:36.319990    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c202cb-89eb-492e-a328-656477c0c60a" path="/var/lib/kubelet/pods/38c202cb-89eb-492e-a328-656477c0c60a/volumes"
Nov 03 13:28:36.320466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:36.320451    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e9e3c5-0977-4a89-a149-12f93e791504" path="/var/lib/kubelet/pods/82e9e3c5-0977-4a89-a149-12f93e791504/volumes"
Nov 03 13:28:38.416116 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:38.416075    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"]
Nov 03 13:28:43.009507 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:43.009470    4447 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7c192b/agc-cred-29dd648f7d07: secret "agc-cred-29dd648f7d07" not found
Nov 03 13:28:43.009507 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:43.009506    4447 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7c192b/agc-cred-f5ffe280c5d9: secret "agc-cred-f5ffe280c5d9" not found
Nov 03 13:28:43.010081 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:43.009535    4447 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n: [secret "agc-cred-29dd648f7d07" not found, secret "agc-cred-f5ffe280c5d9" not found]
Nov 03 13:28:43.010081 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:28:43.009608    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials podName:9728df0b-e653-4464-9f46-771b8a107389 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:59.009588084 +0000 UTC m=+1226.821256257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" (UID: "9728df0b-e653-4464-9f46-771b8a107389") : [secret "agc-cred-29dd648f7d07" not found, secret "agc-cred-f5ffe280c5d9" not found]
Nov 03 13:28:43.416936 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.416835    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" podUID="9728df0b-e653-4464-9f46-771b8a107389" containerName="payload-processing" containerID="cri-o://409f7accc51a814f4e8d17d165d4bdebb908f4f6c5077818f88ceb68c8a9b1b3" gracePeriod=25
Nov 03 13:28:43.554961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.554929    4447 generic.go:356] "Generic (PLEG): container finished" podID="9728df0b-e653-4464-9f46-771b8a107389" containerID="409f7accc51a814f4e8d17d165d4bdebb908f4f6c5077818f88ceb68c8a9b1b3" exitCode=0
Nov 03 13:28:43.555111 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.555002    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" event={"ID":"9728df0b-e653-4464-9f46-771b8a107389","Type":"ContainerDied","Data":"409f7accc51a814f4e8d17d165d4bdebb908f4f6c5077818f88ceb68c8a9b1b3"}
Nov 03 13:28:43.728800 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.728776    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:43.821009 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.820975    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-plugins-config-volume\") pod \"9728df0b-e653-4464-9f46-771b8a107389\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") "
Nov 03 13:28:43.821182 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.821128    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-kube-api-access-vpxvr\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-kube-api-access-vpxvr\") pod \"9728df0b-e653-4464-9f46-771b8a107389\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") "
Nov 03 13:28:43.821240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.821215    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-routing-overlay-volume\") pod \"9728df0b-e653-4464-9f46-771b8a107389\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") "
Nov 03 13:28:43.821296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.821276    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials\") pod \"9728df0b-e653-4464-9f46-771b8a107389\" (UID: \"9728df0b-e653-4464-9f46-771b8a107389\") "
Nov 03 13:28:43.821440 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.821411    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-plugins-config-volume" pod "9728df0b-e653-4464-9f46-771b8a107389" (UID: "9728df0b-e653-4464-9f46-771b8a107389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:43.821674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.821646    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-routing-overlay-volume" pod "9728df0b-e653-4464-9f46-771b8a107389" (UID: "9728df0b-e653-4464-9f46-771b8a107389"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:43.821859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.821843    4447 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-routing-overlay-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:43.821906 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.821861    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9728df0b-e653-4464-9f46-771b8a107389-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:43.823332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.823307    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-kube-api-access-vpxvr" pod "9728df0b-e653-4464-9f46-771b8a107389" (UID: "9728df0b-e653-4464-9f46-771b8a107389"). InnerVolumeSpecName "kube-api-access-vpxvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:43.823462 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.823438    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials" pod "9728df0b-e653-4464-9f46-771b8a107389" (UID: "9728df0b-e653-4464-9f46-771b8a107389"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:43.923706 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.923647    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxvr\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-kube-api-access-vpxvr\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:43.923706 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:43.923680    4447 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9728df0b-e653-4464-9f46-771b8a107389-provider-credentials\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:44.563412 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:44.563381    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"
Nov 03 13:28:44.563412 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:44.563395    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n" event={"ID":"9728df0b-e653-4464-9f46-771b8a107389","Type":"ContainerDied","Data":"4abcebe790719f6fd7779b743014513d913f518f9f6acb37df580e4eb30a742a"}
Nov 03 13:28:44.563957 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:44.563454    4447 scope.go:122] "RemoveContainer" containerID="409f7accc51a814f4e8d17d165d4bdebb908f4f6c5077818f88ceb68c8a9b1b3"
Nov 03 13:28:44.585475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:44.585444    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"]
Nov 03 13:28:44.588943 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:44.588922    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75lfr6n"]
Nov 03 13:28:46.349367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:46.347277    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9728df0b-e653-4464-9f46-771b8a107389" path="/var/lib/kubelet/pods/9728df0b-e653-4464-9f46-771b8a107389/volumes"
Nov 03 13:28:58.779719 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.779685    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7089-68f9bf"]
Nov 03 13:28:58.780313 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.780299    4447 cpu_manager.go:414] "containerMap: removing container" podUID="9728df0b-e653-4464-9f46-771b8a107389" containerName="payload-processing"
Nov 03 13:28:58.780383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.780314    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9728df0b-e653-4464-9f46-771b8a107389" containerName="payload-processing"
Nov 03 13:28:58.780383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.780333    4447 cpu_manager.go:414] "containerMap: removing container" podUID="82e9e3c5-0977-4a89-a149-12f93e791504" containerName="maas-api"
Nov 03 13:28:58.780383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.780338    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e9e3c5-0977-4a89-a149-12f93e791504" containerName="maas-api"
Nov 03 13:28:58.780383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.780366    4447 cpu_manager.go:414] "containerMap: removing container" podUID="38c202cb-89eb-492e-a328-656477c0c60a" containerName="payload-processing"
Nov 03 13:28:58.780383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.780371    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c202cb-89eb-492e-a328-656477c0c60a" containerName="payload-processing"
Nov 03 13:28:58.780549 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.780464    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e9e3c5-0977-4a89-a149-12f93e791504" containerName="maas-api"
Nov 03 13:28:58.780549 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.780475    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c202cb-89eb-492e-a328-656477c0c60a" containerName="payload-processing"
Nov 03 13:28:58.780549 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.780490    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="9728df0b-e653-4464-9f46-771b8a107389" containerName="payload-processing"
Nov 03 13:28:58.784276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.784248    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7089-68f9bf"
Nov 03 13:28:58.787328 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.787304    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncr7v\""
Nov 03 13:28:58.788915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.788888    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7089-68f9bf"]
Nov 03 13:28:58.891888 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.891852    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58rrp\" (UniqueName: \"kubernetes.io/projected/c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d-kube-api-access-58rrp\") pod \"test-curl-7089-68f9bf\" (UID: \"c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d\") " pod="opendatahub/test-curl-7089-68f9bf"
Nov 03 13:28:58.993715 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:58.993667    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58rrp\" (UniqueName: \"kubernetes.io/projected/c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d-kube-api-access-58rrp\") pod \"test-curl-7089-68f9bf\" (UID: \"c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d\") " pod="opendatahub/test-curl-7089-68f9bf"
Nov 03 13:28:59.001046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:59.001017    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58rrp\" (UniqueName: \"kubernetes.io/projected/c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d-kube-api-access-58rrp\") pod \"test-curl-7089-68f9bf\" (UID: \"c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d\") " pod="opendatahub/test-curl-7089-68f9bf"
Nov 03 13:28:59.102748 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:59.102671    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7089-68f9bf"
Nov 03 13:28:59.434848 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:59.434821    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7089-68f9bf"]
Nov 03 13:28:59.435759 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:28:59.435732    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3bd5b8d_6df6_4e8d_8c9c_5b0d8892970d.slice/crio-b7310352b83dccefce4b2612942e7736f3007e56cd1e173eb4cfd7d5df94c3c8 WatchSource:0}: Error finding container b7310352b83dccefce4b2612942e7736f3007e56cd1e173eb4cfd7d5df94c3c8: Status 404 returned error can't find the container with id b7310352b83dccefce4b2612942e7736f3007e56cd1e173eb4cfd7d5df94c3c8
Nov 03 13:28:59.692002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:59.691918    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7089-68f9bf" event={"ID":"c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d","Type":"ContainerStarted","Data":"747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878"}
Nov 03 13:28:59.692002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:59.691967    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7089-68f9bf" event={"ID":"c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d","Type":"ContainerStarted","Data":"b7310352b83dccefce4b2612942e7736f3007e56cd1e173eb4cfd7d5df94c3c8"}
Nov 03 13:28:59.713894 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:28:59.713843    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7089-68f9bf" podStartSLOduration=1.713829914 podStartE2EDuration="1.713829914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:59.710782036 +0000 UTC m=+1227.522450254" watchObservedRunningTime="2026-11-03 13:28:59.713829914 +0000 UTC m=+1227.525498101"
Nov 03 13:29:00.379832 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:00.379790    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7089-68f9bf"]
Nov 03 13:29:00.383804 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:00.383775    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7089-68f9bf"]
Nov 03 13:29:01.705363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:01.705311    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7089-68f9bf" podUID="c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d" containerName="test-curl-7089-68f9bf" containerID="cri-o://747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878" gracePeriod=2
Nov 03 13:29:01.707710 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:29:01.707670    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7089-68f9bf\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d" pod="opendatahub/test-curl-7089-68f9bf"
Nov 03 13:29:02.317704 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:29:02.317671    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7089-68f9bf\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d" pod="opendatahub/test-curl-7089-68f9bf"
Nov 03 13:29:03.412042 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:29:03.412006    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:29:03.910395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:03.910370    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7089-68f9bf"
Nov 03 13:29:03.976015 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:03.975936    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d-kube-api-access-58rrp\" (UniqueName: \"kubernetes.io/projected/c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d-kube-api-access-58rrp\") pod \"c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d\" (UID: \"c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d\") "
Nov 03 13:29:03.978053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:03.978022    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d-kube-api-access-58rrp" pod "c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d" (UID: "c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d"). InnerVolumeSpecName "kube-api-access-58rrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:04.077247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:04.077212    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58rrp\" (UniqueName: \"kubernetes.io/projected/c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d-kube-api-access-58rrp\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:04.321012 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:04.320928    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d" path="/var/lib/kubelet/pods/c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d/volumes"
Nov 03 13:29:04.727674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:04.727643    4447 generic.go:356] "Generic (PLEG): container finished" podID="c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d" containerID="747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878" exitCode=137
Nov 03 13:29:04.728136 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:04.727680    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7089-68f9bf"
Nov 03 13:29:04.728136 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:04.727736    4447 scope.go:122] "RemoveContainer" containerID="747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878"
Nov 03 13:29:04.737611 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:04.737590    4447 scope.go:122] "RemoveContainer" containerID="747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878"
Nov 03 13:29:04.737900 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:29:04.737883    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878\": container with ID starting with 747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878 not found: ID does not exist" containerID="747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878"
Nov 03 13:29:04.737948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:04.737912    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878"} err="failed to get container status \"747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878\": rpc error: code = NotFound desc = could not find container \"747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878\": container with ID starting with 747d949fe6f4ca2e58278fbde9e0e2a590ab5c1d5ea549472996964fedb80878 not found: ID does not exist"
Nov 03 13:29:13.476931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.476900    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"]
Nov 03 13:29:13.477563 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.477547    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d" containerName="test-curl-7089-68f9bf"
Nov 03 13:29:13.477563 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.477563    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d" containerName="test-curl-7089-68f9bf"
Nov 03 13:29:13.477686 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.477676    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3bd5b8d-6df6-4e8d-8c9c-5b0d8892970d" containerName="test-curl-7089-68f9bf"
Nov 03 13:29:13.482919 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.482890    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.496287 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.496262    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"]
Nov 03 13:29:13.571538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.571503    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.571717 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.571639    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkfmd\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-kube-api-access-nkfmd\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.571717 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.571682    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.571717 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.571700    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.571866 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.571844    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.571913 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.571877    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.571973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.571948    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.572012 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.571994    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.572157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.572138    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.673463 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673423    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.673663 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673483    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.673663 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673566    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkfmd\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-kube-api-access-nkfmd\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.673663 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673604    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.673663 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673646    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.673878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673729    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.673878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673780    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.673878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673848    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.674037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673902    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.674037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.673972    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.674037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.674015    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.674260 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.674238    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.674322 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.674272    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.674407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.674320    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.676018 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.675996    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.676267 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.676247    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.681151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.681126    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkfmd\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-kube-api-access-nkfmd\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.681330 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.681311    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:13.798625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:13.798535    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:14.093275 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:14.093181    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"]
Nov 03 13:29:14.098298 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:29:14.098270    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84bdc1b_5569_46f1_811a_2fe1ce4597b6.slice/crio-56b61320baa810675ec047f4a80bbd18e6dbb4ba886d552997eb15a4250af45f WatchSource:0}: Error finding container 56b61320baa810675ec047f4a80bbd18e6dbb4ba886d552997eb15a4250af45f: Status 404 returned error can't find the container with id 56b61320baa810675ec047f4a80bbd18e6dbb4ba886d552997eb15a4250af45f
Nov 03 13:29:14.101520 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:14.101483    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:29:14.101630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:14.101563    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:29:14.101630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:14.101593    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:29:14.866683 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:14.866638    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" event={"ID":"a84bdc1b-5569-46f1-811a-2fe1ce4597b6","Type":"ContainerStarted","Data":"5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292"}
Nov 03 13:29:14.867161 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:14.866708    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" event={"ID":"a84bdc1b-5569-46f1-811a-2fe1ce4597b6","Type":"ContainerStarted","Data":"56b61320baa810675ec047f4a80bbd18e6dbb4ba886d552997eb15a4250af45f"}
Nov 03 13:29:14.890923 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:14.890863    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" podStartSLOduration=1.890848882 podStartE2EDuration="1.890848882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:14.887952524 +0000 UTC m=+1242.699620738" watchObservedRunningTime="2026-11-03 13:29:14.890848882 +0000 UTC m=+1242.702517143"
Nov 03 13:29:15.169001 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.168921    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"]
Nov 03 13:29:15.173577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.173543    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.181376 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.180584    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-d522b5d8feb5\""
Nov 03 13:29:15.182570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.182546    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"]
Nov 03 13:29:15.182695 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.182599    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"openshift-service-ca.crt\""
Nov 03 13:29:15.182695 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.182620    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-e2e--d832c86e-dockercfg-sq45c\""
Nov 03 13:29:15.182695 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.182657    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"routing-overlay\""
Nov 03 13:29:15.182878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.182700    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"kube-root-ca.crt\""
Nov 03 13:29:15.182878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.182714    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-de83083e156c\""
Nov 03 13:29:15.182878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.182702    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-31f9f21f7966\""
Nov 03 13:29:15.182878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.182762    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1a18c8\""
Nov 03 13:29:15.266812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.266769    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.267036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.266920    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.267128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.267057    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkjlf\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-kube-api-access-qkjlf\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.267128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.267091    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.368164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.368129    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.368411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.368234    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkjlf\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-kube-api-access-qkjlf\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.368411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.368255    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.368560 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.368419    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.369146 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.369122    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.369146 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.369137    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.370898 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.370873    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.376197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.376177    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkjlf\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-kube-api-access-qkjlf\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.490522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.490485    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:15.799485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.799409    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:15.801451 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.801425    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 13:29:15.801569 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.801474    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 13:29:15.820701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.820666    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"]
Nov 03 13:29:15.830860 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:29:15.829891    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00380825_e49e_4051_ae95_3a1fece16a7e.slice/crio-925961562c6280638176f32e2c8be7c31ba7e6348dc074ba9c3929074d33c820 WatchSource:0}: Error finding container 925961562c6280638176f32e2c8be7c31ba7e6348dc074ba9c3929074d33c820: Status 404 returned error can't find the container with id 925961562c6280638176f32e2c8be7c31ba7e6348dc074ba9c3929074d33c820
Nov 03 13:29:15.881558 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:15.880607    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc" event={"ID":"00380825-e49e-4051-ae95-3a1fece16a7e","Type":"ContainerStarted","Data":"925961562c6280638176f32e2c8be7c31ba7e6348dc074ba9c3929074d33c820"}
Nov 03 13:29:16.798970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:16.798937    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 13:29:16.799238 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:16.798996    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 13:29:16.890623 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:16.890584    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc" event={"ID":"00380825-e49e-4051-ae95-3a1fece16a7e","Type":"ContainerStarted","Data":"7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783"}
Nov 03 13:29:16.891027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:16.890806    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:16.911388 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:16.911308    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc" podStartSLOduration=1.9112942560000001 podStartE2EDuration="1.911294256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:16.907573177 +0000 UTC m=+1244.719241366" watchObservedRunningTime="2026-11-03 13:29:16.911294256 +0000 UTC m=+1244.722962443"
Nov 03 13:29:17.798914 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:17.798875    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 13:29:17.799130 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:17.798929    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 13:29:18.799059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:18.799021    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 13:29:18.799477 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:18.799080    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 13:29:19.799832 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:19.799798    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 13:29:19.800239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:19.799863    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 13:29:20.799764 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:20.799728    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 13:29:20.799961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:20.799782    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 13:29:21.804502 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:21.804470    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:21.805288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:21.805266    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:21.805986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:21.805970    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:29:22.841495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:22.841458    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"]
Nov 03 13:29:28.469514 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.469483    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"]
Nov 03 13:29:28.472470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.472447    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:28.496731 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.496696    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"]
Nov 03 13:29:28.502279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.502237    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.507296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.507232    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-960bcac1b998\""
Nov 03 13:29:28.513967 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.513941    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"]
Nov 03 13:29:28.636922 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.636876    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.637125 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.637008    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.637280 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.637262    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5nsq\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-kube-api-access-s5nsq\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.637323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.637305    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.739158 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.739060    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.739357 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.739237    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5nsq\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-kube-api-access-s5nsq\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.739357 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.739268    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.739488 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.739371    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.740173 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.740127    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.740288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.740187    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.741912 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.741889    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.746686 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.746654    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5nsq\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-kube-api-access-s5nsq\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.832651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.832613    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:28.965913 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.965875    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"]
Nov 03 13:29:28.972081 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.972052    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:28.978097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.978059    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-5b2047ca80f3\""
Nov 03 13:29:28.978296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.978107    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-5e1bd9e90045\""
Nov 03 13:29:28.979175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.979156    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"openshift-service-ca.crt\""
Nov 03 13:29:28.979412 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.979263    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-e2e--393fd9ed-dockercfg-fnc52\""
Nov 03 13:29:28.979412 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.979304    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-4d65870da959\""
Nov 03 13:29:28.979862 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.979844    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-1c2ab57ceeb2\""
Nov 03 13:29:28.980015 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.979942    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"routing-overlay\""
Nov 03 13:29:28.980015 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.980005    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ef1693\""
Nov 03 13:29:28.996016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:28.995918    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"]
Nov 03 13:29:29.004612 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.004532    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"kube-root-ca.crt\""
Nov 03 13:29:29.117580 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.117554    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"]
Nov 03 13:29:29.118462 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:29:29.118438    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93c6a29_128b_4fcf_949c_3c9a78d5dcf5.slice/crio-a254069cd633859826ce4f12aa4a6aea565c5c36db169b7f8cc38a93db1d948b WatchSource:0}: Error finding container a254069cd633859826ce4f12aa4a6aea565c5c36db169b7f8cc38a93db1d948b: Status 404 returned error can't find the container with id a254069cd633859826ce4f12aa4a6aea565c5c36db169b7f8cc38a93db1d948b
Nov 03 13:29:29.151487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.151455    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.151620 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.151529    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.151677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.151627    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.151783 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.151760    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rn7\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-kube-api-access-n5rn7\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.252998 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.252917    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.253170 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.253031    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rn7\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-kube-api-access-n5rn7\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.253170 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.253138    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.253248 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.253191    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.253901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.253875    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.254028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.253933    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.255768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.255749    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.260561 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.260524    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rn7\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-kube-api-access-n5rn7\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.300772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.300745    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:29.599623 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:29.599591    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"]
Nov 03 13:29:30.011933 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:30.010617    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t" event={"ID":"608fc87f-1493-4fa2-9926-1836e87f0e1c","Type":"ContainerStarted","Data":"2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6"}
Nov 03 13:29:30.011933 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:30.010693    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t" event={"ID":"608fc87f-1493-4fa2-9926-1836e87f0e1c","Type":"ContainerStarted","Data":"83f7780fbc8ca6d56384f988ed24a9f7a400feae344be2e48624c3708985d78d"}
Nov 03 13:29:30.011933 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:30.011869    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:30.016195 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:30.016161    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8" event={"ID":"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5","Type":"ContainerStarted","Data":"e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2"}
Nov 03 13:29:30.016380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:30.016227    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8" event={"ID":"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5","Type":"ContainerStarted","Data":"a254069cd633859826ce4f12aa4a6aea565c5c36db169b7f8cc38a93db1d948b"}
Nov 03 13:29:30.016380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:30.016256    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:30.039873 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:30.039675    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t" podStartSLOduration=2.039615879 podStartE2EDuration="2.039615879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:30.0329132 +0000 UTC m=+1257.844581392" watchObservedRunningTime="2026-11-03 13:29:30.039615879 +0000 UTC m=+1257.851284068"
Nov 03 13:29:30.051136 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:30.051056    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8" podStartSLOduration=2.051037027 podStartE2EDuration="2.051037027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:30.049737191 +0000 UTC m=+1257.861405396" watchObservedRunningTime="2026-11-03 13:29:30.051037027 +0000 UTC m=+1257.862705215"
Nov 03 13:29:33.414515 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:29:33.414474    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:29:33.470416 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.470376    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc" podUID="00380825-e49e-4051-ae95-3a1fece16a7e" containerName="payload-processing" containerID="cri-o://7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783" gracePeriod=25
Nov 03 13:29:33.755159 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.755136    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:33.833184 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.833151    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-kube-api-access-qkjlf\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-kube-api-access-qkjlf\") pod \"00380825-e49e-4051-ae95-3a1fece16a7e\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") "
Nov 03 13:29:33.833502 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.833482    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-plugins-config-volume\") pod \"00380825-e49e-4051-ae95-3a1fece16a7e\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") "
Nov 03 13:29:33.833558 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.833526    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-provider-credentials\") pod \"00380825-e49e-4051-ae95-3a1fece16a7e\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") "
Nov 03 13:29:33.833598 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.833577    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-routing-overlay-volume\") pod \"00380825-e49e-4051-ae95-3a1fece16a7e\" (UID: \"00380825-e49e-4051-ae95-3a1fece16a7e\") "
Nov 03 13:29:33.833839 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.833806    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-plugins-config-volume" pod "00380825-e49e-4051-ae95-3a1fece16a7e" (UID: "00380825-e49e-4051-ae95-3a1fece16a7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:33.833985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.833965    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-routing-overlay-volume" pod "00380825-e49e-4051-ae95-3a1fece16a7e" (UID: "00380825-e49e-4051-ae95-3a1fece16a7e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:33.834263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.834242    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:33.834320 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.834268    4447 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00380825-e49e-4051-ae95-3a1fece16a7e-routing-overlay-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:33.835538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.835515    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-kube-api-access-qkjlf" pod "00380825-e49e-4051-ae95-3a1fece16a7e" (UID: "00380825-e49e-4051-ae95-3a1fece16a7e"). InnerVolumeSpecName "kube-api-access-qkjlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:33.835634 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.835570    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-provider-credentials" pod "00380825-e49e-4051-ae95-3a1fece16a7e" (UID: "00380825-e49e-4051-ae95-3a1fece16a7e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:33.942725 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.942685    4447 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-provider-credentials\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:33.942725 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:33.942725    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkjlf\" (UniqueName: \"kubernetes.io/projected/00380825-e49e-4051-ae95-3a1fece16a7e-kube-api-access-qkjlf\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:34.050966 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.050880    4447 generic.go:356] "Generic (PLEG): container finished" podID="00380825-e49e-4051-ae95-3a1fece16a7e" containerID="7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783" exitCode=0
Nov 03 13:29:34.050966 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.050939    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc" event={"ID":"00380825-e49e-4051-ae95-3a1fece16a7e","Type":"ContainerDied","Data":"7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783"}
Nov 03 13:29:34.050966 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.050943    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"
Nov 03 13:29:34.051239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.050980    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc" event={"ID":"00380825-e49e-4051-ae95-3a1fece16a7e","Type":"ContainerDied","Data":"925961562c6280638176f32e2c8be7c31ba7e6348dc074ba9c3929074d33c820"}
Nov 03 13:29:34.051239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.051002    4447 scope.go:122] "RemoveContainer" containerID="7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783"
Nov 03 13:29:34.064230 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.064203    4447 scope.go:122] "RemoveContainer" containerID="7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783"
Nov 03 13:29:34.064529 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:29:34.064510    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783\": container with ID starting with 7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783 not found: ID does not exist" containerID="7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783"
Nov 03 13:29:34.064603 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.064553    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783"} err="failed to get container status \"7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783\": rpc error: code = NotFound desc = could not find container \"7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783\": container with ID starting with 7875dba0620588298dbfbf66c175a292e01652dacd9922777405821403278783 not found: ID does not exist"
Nov 03 13:29:34.111144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.111112    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"]
Nov 03 13:29:34.119087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.119063    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6dd2m2vc"]
Nov 03 13:29:34.321158 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:34.321061    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00380825-e49e-4051-ae95-3a1fece16a7e" path="/var/lib/kubelet/pods/00380825-e49e-4051-ae95-3a1fece16a7e/volumes"
Nov 03 13:29:37.871152 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:37.871105    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh" podUID="f0562f99-ac88-442d-b17b-86deec063aff" containerName="main" containerID="cri-o://01417c1712addc54801409abb1a8513ab7b1386486bc9415ad7cbbde1f8130c4" gracePeriod=45
Nov 03 13:29:38.087558 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.087526    4447 generic.go:356] "Generic (PLEG): container finished" podID="f0562f99-ac88-442d-b17b-86deec063aff" containerID="01417c1712addc54801409abb1a8513ab7b1386486bc9415ad7cbbde1f8130c4" exitCode=0
Nov 03 13:29:38.087724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.087600    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh" event={"ID":"f0562f99-ac88-442d-b17b-86deec063aff","Type":"ContainerDied","Data":"01417c1712addc54801409abb1a8513ab7b1386486bc9415ad7cbbde1f8130c4"}
Nov 03 13:29:38.182027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.181999    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:29:38.316692 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.316660    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-model-cache\") pod \"f0562f99-ac88-442d-b17b-86deec063aff\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") "
Nov 03 13:29:38.316869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.316748    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-home\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-home\") pod \"f0562f99-ac88-442d-b17b-86deec063aff\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") "
Nov 03 13:29:38.316869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.316826    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0562f99-ac88-442d-b17b-86deec063aff-kube-api-access-h8kvh\" (UniqueName: \"kubernetes.io/projected/f0562f99-ac88-442d-b17b-86deec063aff-kube-api-access-h8kvh\") pod \"f0562f99-ac88-442d-b17b-86deec063aff\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") "
Nov 03 13:29:38.316990 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.316899    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-dshm\") pod \"f0562f99-ac88-442d-b17b-86deec063aff\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") "
Nov 03 13:29:38.316990 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.316973    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-model-cache" pod "f0562f99-ac88-442d-b17b-86deec063aff" (UID: "f0562f99-ac88-442d-b17b-86deec063aff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:38.316990 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.316986    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-tmp-dir\") pod \"f0562f99-ac88-442d-b17b-86deec063aff\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") "
Nov 03 13:29:38.317156 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.317027    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-home" pod "f0562f99-ac88-442d-b17b-86deec063aff" (UID: "f0562f99-ac88-442d-b17b-86deec063aff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:38.317156 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.317140    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0562f99-ac88-442d-b17b-86deec063aff-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0562f99-ac88-442d-b17b-86deec063aff-tls-certs\") pod \"f0562f99-ac88-442d-b17b-86deec063aff\" (UID: \"f0562f99-ac88-442d-b17b-86deec063aff\") "
Nov 03 13:29:38.317323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.317293    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-tmp-dir" pod "f0562f99-ac88-442d-b17b-86deec063aff" (UID: "f0562f99-ac88-442d-b17b-86deec063aff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:38.317775 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.317756    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-tmp-dir\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:38.317881 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.317779    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-model-cache\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:38.317881 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.317791    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-home\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:38.319719 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.319693    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0562f99-ac88-442d-b17b-86deec063aff-tls-certs" pod "f0562f99-ac88-442d-b17b-86deec063aff" (UID: "f0562f99-ac88-442d-b17b-86deec063aff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:29:38.319859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.319839    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0562f99-ac88-442d-b17b-86deec063aff-kube-api-access-h8kvh" pod "f0562f99-ac88-442d-b17b-86deec063aff" (UID: "f0562f99-ac88-442d-b17b-86deec063aff"). InnerVolumeSpecName "kube-api-access-h8kvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:38.319917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.319854    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-dshm" pod "f0562f99-ac88-442d-b17b-86deec063aff" (UID: "f0562f99-ac88-442d-b17b-86deec063aff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:38.419053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.418961    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0562f99-ac88-442d-b17b-86deec063aff-tls-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:38.419053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.419003    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8kvh\" (UniqueName: \"kubernetes.io/projected/f0562f99-ac88-442d-b17b-86deec063aff-kube-api-access-h8kvh\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:38.419053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:38.419021    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0562f99-ac88-442d-b17b-86deec063aff-dshm\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:39.097226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:39.097198    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"
Nov 03 13:29:39.097900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:39.097239    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh" event={"ID":"f0562f99-ac88-442d-b17b-86deec063aff","Type":"ContainerDied","Data":"c3c982cb7417cc8983b06c14858cc1faa3b28708a0b63ac6f029a3256b9af11e"}
Nov 03 13:29:39.097900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:39.097298    4447 scope.go:122] "RemoveContainer" containerID="01417c1712addc54801409abb1a8513ab7b1386486bc9415ad7cbbde1f8130c4"
Nov 03 13:29:39.117913 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:39.117885    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"]
Nov 03 13:29:39.121476 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:39.121454    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh"]
Nov 03 13:29:39.158524 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:39.158486    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/auth-test-model-8f8aa8-kserve-69b6964f5c-nb2hh" podUID="f0562f99-ac88-442d-b17b-86deec063aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:29:40.320172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:40.320134    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0562f99-ac88-442d-b17b-86deec063aff" path="/var/lib/kubelet/pods/f0562f99-ac88-442d-b17b-86deec063aff/volumes"
Nov 03 13:29:41.044370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.042567    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:29:41.692697 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.692662    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-877757f87-7znqp"]
Nov 03 13:29:41.693375 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.693357    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f0562f99-ac88-442d-b17b-86deec063aff" containerName="main"
Nov 03 13:29:41.693438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.693375    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0562f99-ac88-442d-b17b-86deec063aff" containerName="main"
Nov 03 13:29:41.693438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.693415    4447 cpu_manager.go:414] "containerMap: removing container" podUID="00380825-e49e-4051-ae95-3a1fece16a7e" containerName="payload-processing"
Nov 03 13:29:41.693438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.693420    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00380825-e49e-4051-ae95-3a1fece16a7e" containerName="payload-processing"
Nov 03 13:29:41.693542 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.693525    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="00380825-e49e-4051-ae95-3a1fece16a7e" containerName="payload-processing"
Nov 03 13:29:41.693542 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.693539    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0562f99-ac88-442d-b17b-86deec063aff" containerName="main"
Nov 03 13:29:41.697536 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.697514    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.702481 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.702458    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 13:29:41.702638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.702572    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 13:29:41.703027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.703004    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 13:29:41.703122 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.703031    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 13:29:41.708761 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.708736    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-877757f87-7znqp"]
Nov 03 13:29:41.726848 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.726819    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3de2312914d9\""
Nov 03 13:29:41.727035 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.726819    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-498b9d9ffa2b\""
Nov 03 13:29:41.727035 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.726834    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1d111babc549\""
Nov 03 13:29:41.727137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.726878    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vx42v\""
Nov 03 13:29:41.775093 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.775057    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-provider-credentials\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.775274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.775186    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdck7\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-kube-api-access-bdck7\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.775417 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.775403    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-routing-overlay-volume\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.775465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.775433    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-plugins-config-volume\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.876582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.876536    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdck7\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-kube-api-access-bdck7\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.876778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.876711    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-routing-overlay-volume\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.876778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.876738    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-plugins-config-volume\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.876917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.876781    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-provider-credentials\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.877411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.877378    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-routing-overlay-volume\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.877529 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.877510    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-plugins-config-volume\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.879394 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.879371    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-provider-credentials\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:41.883942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:41.883919    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdck7\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-kube-api-access-bdck7\") pod \"payload-processing-external-model-877757f87-7znqp\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") " pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:42.014099 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:42.013914    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:42.046436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:42.046245    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:29:42.403191 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:29:42.403165    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479b6cb0_15f5_47a9_9d06_b19a7a2d895f.slice/crio-e5164a1dea2361c33f12926155b4ccd08ae4eadec1e5f1986f87f750c7827ef4 WatchSource:0}: Error finding container e5164a1dea2361c33f12926155b4ccd08ae4eadec1e5f1986f87f750c7827ef4: Status 404 returned error can't find the container with id e5164a1dea2361c33f12926155b4ccd08ae4eadec1e5f1986f87f750c7827ef4
Nov 03 13:29:42.403568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:42.403537    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-877757f87-7znqp"]
Nov 03 13:29:43.177420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:43.175445    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp" event={"ID":"479b6cb0-15f5-47a9-9d06-b19a7a2d895f","Type":"ContainerStarted","Data":"57a33410cd3706822d2388e68a42b8566c473d550dd6ec119fa4f71ea1613bff"}
Nov 03 13:29:43.177420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:43.175564    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp" event={"ID":"479b6cb0-15f5-47a9-9d06-b19a7a2d895f","Type":"ContainerStarted","Data":"e5164a1dea2361c33f12926155b4ccd08ae4eadec1e5f1986f87f750c7827ef4"}
Nov 03 13:29:43.177420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:43.176949    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:29:43.197659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:43.197603    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp" podStartSLOduration=2.197591004 podStartE2EDuration="2.197591004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:43.193825369 +0000 UTC m=+1271.005493552" watchObservedRunningTime="2026-11-03 13:29:43.197591004 +0000 UTC m=+1271.009259191"
Nov 03 13:29:55.194424 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:29:55.194382    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:30:00.208757 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.206872    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"]
Nov 03 13:30:00.215055 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.215027    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:00.240043 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.236736    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"]
Nov 03 13:30:00.242795 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.242363    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:00.262039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.250438    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"]
Nov 03 13:30:00.262039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.255950    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:00.277552 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.277498    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:30:00.291472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.291442    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"]
Nov 03 13:30:00.295176 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.295139    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"]
Nov 03 13:30:00.299955 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.299930    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"]
Nov 03 13:30:00.304420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.304387    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"]
Nov 03 13:30:00.309952 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.309930    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"]
Nov 03 13:30:00.310073 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.310034    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:00.345263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.344883    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b83265-83ad-4ecb-bce0-631d453db1fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk\" (UID: \"68b83265-83ad-4ecb-bce0-631d453db1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:00.345263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.345082    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhrx\" (UniqueName: \"kubernetes.io/projected/68b83265-83ad-4ecb-bce0-631d453db1fe-kube-api-access-cxhrx\") pod \"maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk\" (UID: \"68b83265-83ad-4ecb-bce0-631d453db1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:00.446288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.446244    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c2b332-d2b7-4725-8f15-9df83a2e8322-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9\" (UID: \"91c2b332-d2b7-4725-8f15-9df83a2e8322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:00.446798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.446775    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b83265-83ad-4ecb-bce0-631d453db1fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk\" (UID: \"68b83265-83ad-4ecb-bce0-631d453db1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:00.446920 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.446826    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfnlp\" (UniqueName: \"kubernetes.io/projected/dfeff074-799d-4342-be32-7d16639582aa-kube-api-access-cfnlp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg\" (UID: \"dfeff074-799d-4342-be32-7d16639582aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:00.446981 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.446927    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhrx\" (UniqueName: \"kubernetes.io/projected/68b83265-83ad-4ecb-bce0-631d453db1fe-kube-api-access-cxhrx\") pod \"maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk\" (UID: \"68b83265-83ad-4ecb-bce0-631d453db1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:00.447044 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.447024    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeff074-799d-4342-be32-7d16639582aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg\" (UID: \"dfeff074-799d-4342-be32-7d16639582aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:00.447112 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.447064    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlzl\" (UniqueName: \"kubernetes.io/projected/91c2b332-d2b7-4725-8f15-9df83a2e8322-kube-api-access-shlzl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9\" (UID: \"91c2b332-d2b7-4725-8f15-9df83a2e8322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:00.447112 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.447091    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc2kb\" (UniqueName: \"kubernetes.io/projected/c72c0e7c-ba47-4329-92cf-034ab65661a3-kube-api-access-wc2kb\") pod \"maas-api-key-cleanup-29895210-d4b5w\" (UID: \"c72c0e7c-ba47-4329-92cf-034ab65661a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:00.447212 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.447195    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c0e7c-ba47-4329-92cf-034ab65661a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895210-d4b5w\" (UID: \"c72c0e7c-ba47-4329-92cf-034ab65661a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:00.447579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.447560    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b83265-83ad-4ecb-bce0-631d453db1fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk\" (UID: \"68b83265-83ad-4ecb-bce0-631d453db1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:00.458616 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.458590    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhrx\" (UniqueName: \"kubernetes.io/projected/68b83265-83ad-4ecb-bce0-631d453db1fe-kube-api-access-cxhrx\") pod \"maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk\" (UID: \"68b83265-83ad-4ecb-bce0-631d453db1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:00.549205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.549106    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c2b332-d2b7-4725-8f15-9df83a2e8322-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9\" (UID: \"91c2b332-d2b7-4725-8f15-9df83a2e8322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:00.549438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.549227    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfnlp\" (UniqueName: \"kubernetes.io/projected/dfeff074-799d-4342-be32-7d16639582aa-kube-api-access-cfnlp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg\" (UID: \"dfeff074-799d-4342-be32-7d16639582aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:00.549438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.549391    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeff074-799d-4342-be32-7d16639582aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg\" (UID: \"dfeff074-799d-4342-be32-7d16639582aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:00.549438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.549415    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlzl\" (UniqueName: \"kubernetes.io/projected/91c2b332-d2b7-4725-8f15-9df83a2e8322-kube-api-access-shlzl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9\" (UID: \"91c2b332-d2b7-4725-8f15-9df83a2e8322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:00.549438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.549438    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc2kb\" (UniqueName: \"kubernetes.io/projected/c72c0e7c-ba47-4329-92cf-034ab65661a3-kube-api-access-wc2kb\") pod \"maas-api-key-cleanup-29895210-d4b5w\" (UID: \"c72c0e7c-ba47-4329-92cf-034ab65661a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:00.549699 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.549479    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c0e7c-ba47-4329-92cf-034ab65661a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895210-d4b5w\" (UID: \"c72c0e7c-ba47-4329-92cf-034ab65661a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:00.549944 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.549920    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c2b332-d2b7-4725-8f15-9df83a2e8322-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9\" (UID: \"91c2b332-d2b7-4725-8f15-9df83a2e8322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:00.550201 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.550178    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeff074-799d-4342-be32-7d16639582aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg\" (UID: \"dfeff074-799d-4342-be32-7d16639582aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:00.550282 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.550262    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c0e7c-ba47-4329-92cf-034ab65661a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895210-d4b5w\" (UID: \"c72c0e7c-ba47-4329-92cf-034ab65661a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:00.557149 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.557121    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlzl\" (UniqueName: \"kubernetes.io/projected/91c2b332-d2b7-4725-8f15-9df83a2e8322-kube-api-access-shlzl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9\" (UID: \"91c2b332-d2b7-4725-8f15-9df83a2e8322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:00.557266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.557151    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc2kb\" (UniqueName: \"kubernetes.io/projected/c72c0e7c-ba47-4329-92cf-034ab65661a3-kube-api-access-wc2kb\") pod \"maas-api-key-cleanup-29895210-d4b5w\" (UID: \"c72c0e7c-ba47-4329-92cf-034ab65661a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:00.557581 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.557562    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfnlp\" (UniqueName: \"kubernetes.io/projected/dfeff074-799d-4342-be32-7d16639582aa-kube-api-access-cfnlp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg\" (UID: \"dfeff074-799d-4342-be32-7d16639582aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:00.567520 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.567495    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:00.583299 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.583076    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:00.591561 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.591533    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:00.654974 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.654599    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:00.878778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.878745    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"]
Nov 03 13:30:00.881554 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:30:00.881503    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b83265_83ad_4ecb_bce0_631d453db1fe.slice/crio-752f4a14a18e88d41ba4c1f30d32fd85a48f385cf46d59f8ae70b156ec43cc29 WatchSource:0}: Error finding container 752f4a14a18e88d41ba4c1f30d32fd85a48f385cf46d59f8ae70b156ec43cc29: Status 404 returned error can't find the container with id 752f4a14a18e88d41ba4c1f30d32fd85a48f385cf46d59f8ae70b156ec43cc29
Nov 03 13:30:00.909506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.909479    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"]
Nov 03 13:30:00.915699 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:30:00.915673    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfeff074_799d_4342_be32_7d16639582aa.slice/crio-ea66c5c1c68a42bfc754ed234c8106cc4fef9a7fb8123f5e749d44b5cc3e5119 WatchSource:0}: Error finding container ea66c5c1c68a42bfc754ed234c8106cc4fef9a7fb8123f5e749d44b5cc3e5119: Status 404 returned error can't find the container with id ea66c5c1c68a42bfc754ed234c8106cc4fef9a7fb8123f5e749d44b5cc3e5119
Nov 03 13:30:00.933674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:00.933645    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"]
Nov 03 13:30:00.934080 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:30:00.934057    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c2b332_d2b7_4725_8f15_9df83a2e8322.slice/crio-de3e2d25de47d482325fff37b01393dfdde18c029b3650f9c44aeb8b4d39c538 WatchSource:0}: Error finding container de3e2d25de47d482325fff37b01393dfdde18c029b3650f9c44aeb8b4d39c538: Status 404 returned error can't find the container with id de3e2d25de47d482325fff37b01393dfdde18c029b3650f9c44aeb8b4d39c538
Nov 03 13:30:01.015477 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:01.015446    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"]
Nov 03 13:30:01.038177 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:30:01.038137    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72c0e7c_ba47_4329_92cf_034ab65661a3.slice/crio-423ba501497b942f3ece35c4e12fa8449444ef7e6b36a4d89c9978a3d06611df WatchSource:0}: Error finding container 423ba501497b942f3ece35c4e12fa8449444ef7e6b36a4d89c9978a3d06611df: Status 404 returned error can't find the container with id 423ba501497b942f3ece35c4e12fa8449444ef7e6b36a4d89c9978a3d06611df
Nov 03 13:30:01.358554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:01.358520    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" event={"ID":"91c2b332-d2b7-4725-8f15-9df83a2e8322","Type":"ContainerStarted","Data":"de3e2d25de47d482325fff37b01393dfdde18c029b3650f9c44aeb8b4d39c538"}
Nov 03 13:30:01.360039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:01.360011    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" event={"ID":"68b83265-83ad-4ecb-bce0-631d453db1fe","Type":"ContainerStarted","Data":"dedfae9f727cd6bd862fd0d827e370a529c6d99d57d62404eb55ca4d07f6caec"}
Nov 03 13:30:01.360183 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:01.360064    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" event={"ID":"68b83265-83ad-4ecb-bce0-631d453db1fe","Type":"ContainerStarted","Data":"752f4a14a18e88d41ba4c1f30d32fd85a48f385cf46d59f8ae70b156ec43cc29"}
Nov 03 13:30:01.361862 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:01.361836    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" event={"ID":"dfeff074-799d-4342-be32-7d16639582aa","Type":"ContainerStarted","Data":"347fd076965e2c2578304d0cc88fc1cc0a2a6d2605f2a55bbad849ebbc1d3e1c"}
Nov 03 13:30:01.361953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:01.361884    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" event={"ID":"dfeff074-799d-4342-be32-7d16639582aa","Type":"ContainerStarted","Data":"ea66c5c1c68a42bfc754ed234c8106cc4fef9a7fb8123f5e749d44b5cc3e5119"}
Nov 03 13:30:01.363133 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:01.363112    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" event={"ID":"c72c0e7c-ba47-4329-92cf-034ab65661a3","Type":"ContainerStarted","Data":"423ba501497b942f3ece35c4e12fa8449444ef7e6b36a4d89c9978a3d06611df"}
Nov 03 13:30:01.398371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:01.398305    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" podStartSLOduration=1.398292145 podStartE2EDuration="1.398292145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:01.396774099 +0000 UTC m=+1289.208442311" watchObservedRunningTime="2026-11-03 13:30:01.398292145 +0000 UTC m=+1289.209960332"
Nov 03 13:30:01.425685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:01.425610    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" podStartSLOduration=1.425590841 podStartE2EDuration="1.425590841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:01.425022027 +0000 UTC m=+1289.236690239" watchObservedRunningTime="2026-11-03 13:30:01.425590841 +0000 UTC m=+1289.237259031"
Nov 03 13:30:02.372418 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.372371    4447 generic.go:356] "Generic (PLEG): container finished" podID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerID="dedfae9f727cd6bd862fd0d827e370a529c6d99d57d62404eb55ca4d07f6caec" exitCode=60
Nov 03 13:30:02.372418 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.372406    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" event={"ID":"68b83265-83ad-4ecb-bce0-631d453db1fe","Type":"ContainerDied","Data":"dedfae9f727cd6bd862fd0d827e370a529c6d99d57d62404eb55ca4d07f6caec"}
Nov 03 13:30:02.372977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.372811    4447 scope.go:122] "RemoveContainer" containerID="dedfae9f727cd6bd862fd0d827e370a529c6d99d57d62404eb55ca4d07f6caec"
Nov 03 13:30:02.374270 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.374245    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfeff074-799d-4342-be32-7d16639582aa" containerID="347fd076965e2c2578304d0cc88fc1cc0a2a6d2605f2a55bbad849ebbc1d3e1c" exitCode=60
Nov 03 13:30:02.374403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.374332    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" event={"ID":"dfeff074-799d-4342-be32-7d16639582aa","Type":"ContainerDied","Data":"347fd076965e2c2578304d0cc88fc1cc0a2a6d2605f2a55bbad849ebbc1d3e1c"}
Nov 03 13:30:02.374675 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.374601    4447 scope.go:122] "RemoveContainer" containerID="347fd076965e2c2578304d0cc88fc1cc0a2a6d2605f2a55bbad849ebbc1d3e1c"
Nov 03 13:30:02.376555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.376511    4447 generic.go:356] "Generic (PLEG): container finished" podID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerID="1022701fbf0b1c1794a1784d64b92177ba2b850151a9807276b489a66ec290e7" exitCode=60
Nov 03 13:30:02.377059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.376659    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" event={"ID":"c72c0e7c-ba47-4329-92cf-034ab65661a3","Type":"ContainerDied","Data":"1022701fbf0b1c1794a1784d64b92177ba2b850151a9807276b489a66ec290e7"}
Nov 03 13:30:02.377059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.376813    4447 scope.go:122] "RemoveContainer" containerID="1022701fbf0b1c1794a1784d64b92177ba2b850151a9807276b489a66ec290e7"
Nov 03 13:30:02.378498 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.378478    4447 generic.go:356] "Generic (PLEG): container finished" podID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerID="b6e89e2ee798fe2586f0e2dd22e8f4855d80e351054a64fc120fff7bfde4ec6a" exitCode=60
Nov 03 13:30:02.378573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.378532    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" event={"ID":"91c2b332-d2b7-4725-8f15-9df83a2e8322","Type":"ContainerDied","Data":"b6e89e2ee798fe2586f0e2dd22e8f4855d80e351054a64fc120fff7bfde4ec6a"}
Nov 03 13:30:02.378708 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:02.378694    4447 scope.go:122] "RemoveContainer" containerID="b6e89e2ee798fe2586f0e2dd22e8f4855d80e351054a64fc120fff7bfde4ec6a"
Nov 03 13:30:03.387900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:03.387809    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" event={"ID":"68b83265-83ad-4ecb-bce0-631d453db1fe","Type":"ContainerStarted","Data":"a682269388e1ee7866a721f9332e1eb2fcdbce7b644701ad5f4218e1d43635d8"}
Nov 03 13:30:03.389868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:03.389839    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" event={"ID":"dfeff074-799d-4342-be32-7d16639582aa","Type":"ContainerStarted","Data":"4d6598d0f7040a3c38728ace8436550f23a53bf34e8c2a95300cd6723087fd3a"}
Nov 03 13:30:03.391981 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:03.391953    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" event={"ID":"c72c0e7c-ba47-4329-92cf-034ab65661a3","Type":"ContainerStarted","Data":"57a2cd0b3bef97c6aa83383af7776f6569721fcb1738fb4e94e75dfb02d6a8cc"}
Nov 03 13:30:03.393855 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:03.393832    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" event={"ID":"91c2b332-d2b7-4725-8f15-9df83a2e8322","Type":"ContainerStarted","Data":"10bb710adb472b208c80264d9a65c14a64ef687f74661e255c969bf6883fe246"}
Nov 03 13:30:03.414788 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:03.414749    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:30:03.436019 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:03.435951    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" podStartSLOduration=3.435931403 podStartE2EDuration="3.435931403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:03.432294863 +0000 UTC m=+1291.243963052" watchObservedRunningTime="2026-11-03 13:30:03.435931403 +0000 UTC m=+1291.247599591"
Nov 03 13:30:03.448870 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:03.448821    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" podStartSLOduration=3.448807091 podStartE2EDuration="3.448807091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:03.446364082 +0000 UTC m=+1291.258032271" watchObservedRunningTime="2026-11-03 13:30:03.448807091 +0000 UTC m=+1291.260475280"
Nov 03 13:30:04.404049 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.404015    4447 generic.go:356] "Generic (PLEG): container finished" podID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerID="10bb710adb472b208c80264d9a65c14a64ef687f74661e255c969bf6883fe246" exitCode=60
Nov 03 13:30:04.404561 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.404095    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" event={"ID":"91c2b332-d2b7-4725-8f15-9df83a2e8322","Type":"ContainerDied","Data":"10bb710adb472b208c80264d9a65c14a64ef687f74661e255c969bf6883fe246"}
Nov 03 13:30:04.404561 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.404166    4447 scope.go:122] "RemoveContainer" containerID="b6e89e2ee798fe2586f0e2dd22e8f4855d80e351054a64fc120fff7bfde4ec6a"
Nov 03 13:30:04.404561 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.404468    4447 scope.go:122] "RemoveContainer" containerID="10bb710adb472b208c80264d9a65c14a64ef687f74661e255c969bf6883fe246"
Nov 03 13:30:04.404879 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:04.404830    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9_odh-ai-gateway-infra(91c2b332-d2b7-4725-8f15-9df83a2e8322)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322"
Nov 03 13:30:04.406265 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.406243    4447 generic.go:356] "Generic (PLEG): container finished" podID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerID="a682269388e1ee7866a721f9332e1eb2fcdbce7b644701ad5f4218e1d43635d8" exitCode=60
Nov 03 13:30:04.406382 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.406267    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" event={"ID":"68b83265-83ad-4ecb-bce0-631d453db1fe","Type":"ContainerDied","Data":"a682269388e1ee7866a721f9332e1eb2fcdbce7b644701ad5f4218e1d43635d8"}
Nov 03 13:30:04.406598 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.406584    4447 scope.go:122] "RemoveContainer" containerID="a682269388e1ee7866a721f9332e1eb2fcdbce7b644701ad5f4218e1d43635d8"
Nov 03 13:30:04.406924 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:04.406903    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk_odh-ai-gateway-infra(68b83265-83ad-4ecb-bce0-631d453db1fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe"
Nov 03 13:30:04.408205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.408182    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfeff074-799d-4342-be32-7d16639582aa" containerID="4d6598d0f7040a3c38728ace8436550f23a53bf34e8c2a95300cd6723087fd3a" exitCode=60
Nov 03 13:30:04.408358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.408289    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" event={"ID":"dfeff074-799d-4342-be32-7d16639582aa","Type":"ContainerDied","Data":"4d6598d0f7040a3c38728ace8436550f23a53bf34e8c2a95300cd6723087fd3a"}
Nov 03 13:30:04.408874 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.408850    4447 scope.go:122] "RemoveContainer" containerID="4d6598d0f7040a3c38728ace8436550f23a53bf34e8c2a95300cd6723087fd3a"
Nov 03 13:30:04.409293 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:04.409251    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg_odh-ai-gateway-infra(dfeff074-799d-4342-be32-7d16639582aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" podUID="dfeff074-799d-4342-be32-7d16639582aa"
Nov 03 13:30:04.410821 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.410796    4447 generic.go:356] "Generic (PLEG): container finished" podID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerID="57a2cd0b3bef97c6aa83383af7776f6569721fcb1738fb4e94e75dfb02d6a8cc" exitCode=60
Nov 03 13:30:04.410919 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.410836    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" event={"ID":"c72c0e7c-ba47-4329-92cf-034ab65661a3","Type":"ContainerDied","Data":"57a2cd0b3bef97c6aa83383af7776f6569721fcb1738fb4e94e75dfb02d6a8cc"}
Nov 03 13:30:04.411107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.411093    4447 scope.go:122] "RemoveContainer" containerID="57a2cd0b3bef97c6aa83383af7776f6569721fcb1738fb4e94e75dfb02d6a8cc"
Nov 03 13:30:04.411428 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:04.411407    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895210-d4b5w_odh-ai-gateway-infra(c72c0e7c-ba47-4329-92cf-034ab65661a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3"
Nov 03 13:30:04.419581 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.419564    4447 scope.go:122] "RemoveContainer" containerID="dedfae9f727cd6bd862fd0d827e370a529c6d99d57d62404eb55ca4d07f6caec"
Nov 03 13:30:04.440712 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.440689    4447 scope.go:122] "RemoveContainer" containerID="347fd076965e2c2578304d0cc88fc1cc0a2a6d2605f2a55bbad849ebbc1d3e1c"
Nov 03 13:30:04.455076 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:04.455043    4447 scope.go:122] "RemoveContainer" containerID="1022701fbf0b1c1794a1784d64b92177ba2b850151a9807276b489a66ec290e7"
Nov 03 13:30:05.421198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:05.421160    4447 scope.go:122] "RemoveContainer" containerID="a682269388e1ee7866a721f9332e1eb2fcdbce7b644701ad5f4218e1d43635d8"
Nov 03 13:30:05.421702 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:05.421559    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk_odh-ai-gateway-infra(68b83265-83ad-4ecb-bce0-631d453db1fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe"
Nov 03 13:30:05.422912 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:05.422888    4447 scope.go:122] "RemoveContainer" containerID="4d6598d0f7040a3c38728ace8436550f23a53bf34e8c2a95300cd6723087fd3a"
Nov 03 13:30:05.423298 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:05.423271    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg_odh-ai-gateway-infra(dfeff074-799d-4342-be32-7d16639582aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" podUID="dfeff074-799d-4342-be32-7d16639582aa"
Nov 03 13:30:05.424466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:05.424444    4447 scope.go:122] "RemoveContainer" containerID="57a2cd0b3bef97c6aa83383af7776f6569721fcb1738fb4e94e75dfb02d6a8cc"
Nov 03 13:30:05.424763 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:05.424742    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895210-d4b5w_odh-ai-gateway-infra(c72c0e7c-ba47-4329-92cf-034ab65661a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3"
Nov 03 13:30:05.426597 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:05.426578    4447 scope.go:122] "RemoveContainer" containerID="10bb710adb472b208c80264d9a65c14a64ef687f74661e255c969bf6883fe246"
Nov 03 13:30:05.426874 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:05.426854    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9_odh-ai-gateway-infra(91c2b332-d2b7-4725-8f15-9df83a2e8322)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322"
Nov 03 13:30:14.313074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:14.313037    4447 scope.go:122] "RemoveContainer" containerID="57a2cd0b3bef97c6aa83383af7776f6569721fcb1738fb4e94e75dfb02d6a8cc"
Nov 03 13:30:14.313617 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:14.313402    4447 scope.go:122] "RemoveContainer" containerID="4d6598d0f7040a3c38728ace8436550f23a53bf34e8c2a95300cd6723087fd3a"
Nov 03 13:30:14.314810 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:14.313920    4447 scope.go:122] "RemoveContainer" containerID="a682269388e1ee7866a721f9332e1eb2fcdbce7b644701ad5f4218e1d43635d8"
Nov 03 13:30:15.513848 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:15.513808    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" event={"ID":"68b83265-83ad-4ecb-bce0-631d453db1fe","Type":"ContainerStarted","Data":"d92764033be55fd86a92d6492e5223ec5a43f5fb3cadb9ea7dcea21477073dad"}
Nov 03 13:30:15.516216 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:15.516187    4447 generic.go:356] "Generic (PLEG): container finished" podID="dfeff074-799d-4342-be32-7d16639582aa" containerID="8c9c881b0b706e335d162e02997bc87b09b0a114a1dfa72e61de9e2186880456" exitCode=60
Nov 03 13:30:15.516374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:15.516240    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" event={"ID":"dfeff074-799d-4342-be32-7d16639582aa","Type":"ContainerDied","Data":"8c9c881b0b706e335d162e02997bc87b09b0a114a1dfa72e61de9e2186880456"}
Nov 03 13:30:15.516374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:15.516296    4447 scope.go:122] "RemoveContainer" containerID="4d6598d0f7040a3c38728ace8436550f23a53bf34e8c2a95300cd6723087fd3a"
Nov 03 13:30:15.516675 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:15.516654    4447 scope.go:122] "RemoveContainer" containerID="8c9c881b0b706e335d162e02997bc87b09b0a114a1dfa72e61de9e2186880456"
Nov 03 13:30:15.516978 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:15.516958    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg_odh-ai-gateway-infra(dfeff074-799d-4342-be32-7d16639582aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" podUID="dfeff074-799d-4342-be32-7d16639582aa"
Nov 03 13:30:15.518511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:15.518489    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" event={"ID":"c72c0e7c-ba47-4329-92cf-034ab65661a3","Type":"ContainerStarted","Data":"76143869e9bcdc33180a275d2d4beb3bff9f03b8f8d7c29cf90e8ec0f2c88c6c"}
Nov 03 13:30:16.529183 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.529150    4447 generic.go:356] "Generic (PLEG): container finished" podID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerID="d92764033be55fd86a92d6492e5223ec5a43f5fb3cadb9ea7dcea21477073dad" exitCode=60
Nov 03 13:30:16.529667 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.529223    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" event={"ID":"68b83265-83ad-4ecb-bce0-631d453db1fe","Type":"ContainerDied","Data":"d92764033be55fd86a92d6492e5223ec5a43f5fb3cadb9ea7dcea21477073dad"}
Nov 03 13:30:16.529667 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.529280    4447 scope.go:122] "RemoveContainer" containerID="a682269388e1ee7866a721f9332e1eb2fcdbce7b644701ad5f4218e1d43635d8"
Nov 03 13:30:16.529834 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.529816    4447 scope.go:122] "RemoveContainer" containerID="d92764033be55fd86a92d6492e5223ec5a43f5fb3cadb9ea7dcea21477073dad"
Nov 03 13:30:16.530672 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:16.530260    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk_odh-ai-gateway-infra(68b83265-83ad-4ecb-bce0-631d453db1fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe"
Nov 03 13:30:16.533719 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.533688    4447 generic.go:356] "Generic (PLEG): container finished" podID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerID="76143869e9bcdc33180a275d2d4beb3bff9f03b8f8d7c29cf90e8ec0f2c88c6c" exitCode=60
Nov 03 13:30:16.533824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.533756    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" event={"ID":"c72c0e7c-ba47-4329-92cf-034ab65661a3","Type":"ContainerDied","Data":"76143869e9bcdc33180a275d2d4beb3bff9f03b8f8d7c29cf90e8ec0f2c88c6c"}
Nov 03 13:30:16.534204 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.534189    4447 scope.go:122] "RemoveContainer" containerID="76143869e9bcdc33180a275d2d4beb3bff9f03b8f8d7c29cf90e8ec0f2c88c6c"
Nov 03 13:30:16.534562 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:16.534544    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895210-d4b5w_odh-ai-gateway-infra(c72c0e7c-ba47-4329-92cf-034ab65661a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3"
Nov 03 13:30:16.543407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.543383    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"]
Nov 03 13:30:16.550740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.550698    4447 scope.go:122] "RemoveContainer" containerID="57a2cd0b3bef97c6aa83383af7776f6569721fcb1738fb4e94e75dfb02d6a8cc"
Nov 03 13:30:16.582452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.578884    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"]
Nov 03 13:30:16.601060 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.601015    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"]
Nov 03 13:30:16.806930 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.806901    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:16.871297 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.871252    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfeff074-799d-4342-be32-7d16639582aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeff074-799d-4342-be32-7d16639582aa-maas-api-service-ca\") pod \"dfeff074-799d-4342-be32-7d16639582aa\" (UID: \"dfeff074-799d-4342-be32-7d16639582aa\") "
Nov 03 13:30:16.871488 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.871420    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfeff074-799d-4342-be32-7d16639582aa-kube-api-access-cfnlp\" (UniqueName: \"kubernetes.io/projected/dfeff074-799d-4342-be32-7d16639582aa-kube-api-access-cfnlp\") pod \"dfeff074-799d-4342-be32-7d16639582aa\" (UID: \"dfeff074-799d-4342-be32-7d16639582aa\") "
Nov 03 13:30:16.872455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.872425    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfeff074-799d-4342-be32-7d16639582aa-maas-api-service-ca" pod "dfeff074-799d-4342-be32-7d16639582aa" (UID: "dfeff074-799d-4342-be32-7d16639582aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:16.873986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.873956    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfeff074-799d-4342-be32-7d16639582aa-kube-api-access-cfnlp" pod "dfeff074-799d-4342-be32-7d16639582aa" (UID: "dfeff074-799d-4342-be32-7d16639582aa"). InnerVolumeSpecName "kube-api-access-cfnlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:16.973454 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.973412    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeff074-799d-4342-be32-7d16639582aa-maas-api-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:16.973454 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:16.973446    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfnlp\" (UniqueName: \"kubernetes.io/projected/dfeff074-799d-4342-be32-7d16639582aa-kube-api-access-cfnlp\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:17.547378 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.547307    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"
Nov 03 13:30:17.553726 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.553405    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg" event={"ID":"dfeff074-799d-4342-be32-7d16639582aa","Type":"ContainerDied","Data":"ea66c5c1c68a42bfc754ed234c8106cc4fef9a7fb8123f5e749d44b5cc3e5119"}
Nov 03 13:30:17.553726 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.553484    4447 scope.go:122] "RemoveContainer" containerID="8c9c881b0b706e335d162e02997bc87b09b0a114a1dfa72e61de9e2186880456"
Nov 03 13:30:17.604400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.602240    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"]
Nov 03 13:30:17.608240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.608207    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9add72-29895210-9hgbg"]
Nov 03 13:30:17.816904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.816874    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:17.881023 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.880996    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:17.909958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.909916    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b83265-83ad-4ecb-bce0-631d453db1fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b83265-83ad-4ecb-bce0-631d453db1fe-maas-api-service-ca\") pod \"68b83265-83ad-4ecb-bce0-631d453db1fe\" (UID: \"68b83265-83ad-4ecb-bce0-631d453db1fe\") "
Nov 03 13:30:17.910791 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.910727    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b83265-83ad-4ecb-bce0-631d453db1fe-kube-api-access-cxhrx\" (UniqueName: \"kubernetes.io/projected/68b83265-83ad-4ecb-bce0-631d453db1fe-kube-api-access-cxhrx\") pod \"68b83265-83ad-4ecb-bce0-631d453db1fe\" (UID: \"68b83265-83ad-4ecb-bce0-631d453db1fe\") "
Nov 03 13:30:17.911075 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.910422    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b83265-83ad-4ecb-bce0-631d453db1fe-maas-api-service-ca" pod "68b83265-83ad-4ecb-bce0-631d453db1fe" (UID: "68b83265-83ad-4ecb-bce0-631d453db1fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:17.912010 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.911986    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b83265-83ad-4ecb-bce0-631d453db1fe-maas-api-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:17.918844 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:17.918801    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b83265-83ad-4ecb-bce0-631d453db1fe-kube-api-access-cxhrx" pod "68b83265-83ad-4ecb-bce0-631d453db1fe" (UID: "68b83265-83ad-4ecb-bce0-631d453db1fe"). InnerVolumeSpecName "kube-api-access-cxhrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:18.012791 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.012755    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72c0e7c-ba47-4329-92cf-034ab65661a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c0e7c-ba47-4329-92cf-034ab65661a3-maas-api-service-ca\") pod \"c72c0e7c-ba47-4329-92cf-034ab65661a3\" (UID: \"c72c0e7c-ba47-4329-92cf-034ab65661a3\") "
Nov 03 13:30:18.013052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.013031    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72c0e7c-ba47-4329-92cf-034ab65661a3-kube-api-access-wc2kb\" (UniqueName: \"kubernetes.io/projected/c72c0e7c-ba47-4329-92cf-034ab65661a3-kube-api-access-wc2kb\") pod \"c72c0e7c-ba47-4329-92cf-034ab65661a3\" (UID: \"c72c0e7c-ba47-4329-92cf-034ab65661a3\") "
Nov 03 13:30:18.013207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.013180    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72c0e7c-ba47-4329-92cf-034ab65661a3-maas-api-service-ca" pod "c72c0e7c-ba47-4329-92cf-034ab65661a3" (UID: "c72c0e7c-ba47-4329-92cf-034ab65661a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:18.015739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.014131    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxhrx\" (UniqueName: \"kubernetes.io/projected/68b83265-83ad-4ecb-bce0-631d453db1fe-kube-api-access-cxhrx\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:18.015739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.014208    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c0e7c-ba47-4329-92cf-034ab65661a3-maas-api-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:18.015934 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.015896    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72c0e7c-ba47-4329-92cf-034ab65661a3-kube-api-access-wc2kb" pod "c72c0e7c-ba47-4329-92cf-034ab65661a3" (UID: "c72c0e7c-ba47-4329-92cf-034ab65661a3"). InnerVolumeSpecName "kube-api-access-wc2kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:18.115978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.115889    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc2kb\" (UniqueName: \"kubernetes.io/projected/c72c0e7c-ba47-4329-92cf-034ab65661a3-kube-api-access-wc2kb\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:18.339276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.339237    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfeff074-799d-4342-be32-7d16639582aa" path="/var/lib/kubelet/pods/dfeff074-799d-4342-be32-7d16639582aa/volumes"
Nov 03 13:30:18.340908 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.340882    4447 scope.go:122] "RemoveContainer" containerID="10bb710adb472b208c80264d9a65c14a64ef687f74661e255c969bf6883fe246"
Nov 03 13:30:18.568068 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.568042    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"
Nov 03 13:30:18.568068 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.568059    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk" event={"ID":"68b83265-83ad-4ecb-bce0-631d453db1fe","Type":"ContainerDied","Data":"752f4a14a18e88d41ba4c1f30d32fd85a48f385cf46d59f8ae70b156ec43cc29"}
Nov 03 13:30:18.568599 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.568126    4447 scope.go:122] "RemoveContainer" containerID="d92764033be55fd86a92d6492e5223ec5a43f5fb3cadb9ea7dcea21477073dad"
Nov 03 13:30:18.570964 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.570940    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"
Nov 03 13:30:18.571091 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.570972    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w" event={"ID":"c72c0e7c-ba47-4329-92cf-034ab65661a3","Type":"ContainerDied","Data":"423ba501497b942f3ece35c4e12fa8449444ef7e6b36a4d89c9978a3d06611df"}
Nov 03 13:30:18.580283 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.580261    4447 scope.go:122] "RemoveContainer" containerID="76143869e9bcdc33180a275d2d4beb3bff9f03b8f8d7c29cf90e8ec0f2c88c6c"
Nov 03 13:30:18.591979 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.591957    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"]
Nov 03 13:30:18.597929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.597901    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cf53a4-29895210-7j7tk"]
Nov 03 13:30:18.611161 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.611125    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"]
Nov 03 13:30:18.619120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:18.619090    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-d4b5w"]
Nov 03 13:30:19.579622 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:19.579587    4447 generic.go:356] "Generic (PLEG): container finished" podID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerID="b163e8474b6b6790339e282d7146f614ca8b70467d104418999f284c1bc39f72" exitCode=60
Nov 03 13:30:19.580054 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:19.579660    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" event={"ID":"91c2b332-d2b7-4725-8f15-9df83a2e8322","Type":"ContainerDied","Data":"b163e8474b6b6790339e282d7146f614ca8b70467d104418999f284c1bc39f72"}
Nov 03 13:30:19.580054 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:19.579720    4447 scope.go:122] "RemoveContainer" containerID="10bb710adb472b208c80264d9a65c14a64ef687f74661e255c969bf6883fe246"
Nov 03 13:30:19.580174 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:19.580150    4447 scope.go:122] "RemoveContainer" containerID="b163e8474b6b6790339e282d7146f614ca8b70467d104418999f284c1bc39f72"
Nov 03 13:30:19.580589 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:19.580562    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9_odh-ai-gateway-infra(91c2b332-d2b7-4725-8f15-9df83a2e8322)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322"
Nov 03 13:30:20.320938 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:20.320905    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" path="/var/lib/kubelet/pods/68b83265-83ad-4ecb-bce0-631d453db1fe/volumes"
Nov 03 13:30:20.321460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:20.321445    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" path="/var/lib/kubelet/pods/c72c0e7c-ba47-4329-92cf-034ab65661a3/volumes"
Nov 03 13:30:20.621252 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:20.621164    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"]
Nov 03 13:30:20.792938 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:20.792911    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:20.962860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:20.962819    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c2b332-d2b7-4725-8f15-9df83a2e8322-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c2b332-d2b7-4725-8f15-9df83a2e8322-maas-api-service-ca\") pod \"91c2b332-d2b7-4725-8f15-9df83a2e8322\" (UID: \"91c2b332-d2b7-4725-8f15-9df83a2e8322\") "
Nov 03 13:30:20.963078 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:20.962946    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c2b332-d2b7-4725-8f15-9df83a2e8322-kube-api-access-shlzl\" (UniqueName: \"kubernetes.io/projected/91c2b332-d2b7-4725-8f15-9df83a2e8322-kube-api-access-shlzl\") pod \"91c2b332-d2b7-4725-8f15-9df83a2e8322\" (UID: \"91c2b332-d2b7-4725-8f15-9df83a2e8322\") "
Nov 03 13:30:20.963304 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:20.963273    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c2b332-d2b7-4725-8f15-9df83a2e8322-maas-api-service-ca" pod "91c2b332-d2b7-4725-8f15-9df83a2e8322" (UID: "91c2b332-d2b7-4725-8f15-9df83a2e8322"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:20.963627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:20.963604    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c2b332-d2b7-4725-8f15-9df83a2e8322-maas-api-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:20.966208 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:20.966185    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c2b332-d2b7-4725-8f15-9df83a2e8322-kube-api-access-shlzl" pod "91c2b332-d2b7-4725-8f15-9df83a2e8322" (UID: "91c2b332-d2b7-4725-8f15-9df83a2e8322"). InnerVolumeSpecName "kube-api-access-shlzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:21.064627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:21.064591    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shlzl\" (UniqueName: \"kubernetes.io/projected/91c2b332-d2b7-4725-8f15-9df83a2e8322-kube-api-access-shlzl\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:21.599275 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:21.599241    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"
Nov 03 13:30:21.599475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:21.599289    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9" event={"ID":"91c2b332-d2b7-4725-8f15-9df83a2e8322","Type":"ContainerDied","Data":"de3e2d25de47d482325fff37b01393dfdde18c029b3650f9c44aeb8b4d39c538"}
Nov 03 13:30:21.599475 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:21.599381    4447 scope.go:122] "RemoveContainer" containerID="b163e8474b6b6790339e282d7146f614ca8b70467d104418999f284c1bc39f72"
Nov 03 13:30:21.628720 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:21.628687    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"]
Nov 03 13:30:21.639904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:21.639866    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8f8aa8-29895210-9zfk9"]
Nov 03 13:30:22.320843 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:22.320811    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" path="/var/lib/kubelet/pods/91c2b332-d2b7-4725-8f15-9df83a2e8322/volumes"
Nov 03 13:30:33.456696 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:30:33.456649    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:30:33.820677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.820595    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"]
Nov 03 13:30:33.821290 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821274    4447 cpu_manager.go:414] "containerMap: removing container" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821290    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821304    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfeff074-799d-4342-be32-7d16639582aa" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821310    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfeff074-799d-4342-be32-7d16639582aa" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821322    4447 cpu_manager.go:414] "containerMap: removing container" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821327    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821357    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821363    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821374    4447 cpu_manager.go:414] "containerMap: removing container" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821378    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821387    4447 cpu_manager.go:414] "containerMap: removing container" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821391    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821401    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfeff074-799d-4342-be32-7d16639582aa" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821405    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfeff074-799d-4342-be32-7d16639582aa" containerName="cleanup"
Nov 03 13:30:33.821407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821412    4447 cpu_manager.go:414] "containerMap: removing container" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821416    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821425    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821429    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821436    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dfeff074-799d-4342-be32-7d16639582aa" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821439    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfeff074-799d-4342-be32-7d16639582aa" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821541    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821549    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821556    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821562    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821567    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfeff074-799d-4342-be32-7d16639582aa" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821571    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821575    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821579    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfeff074-799d-4342-be32-7d16639582aa" containerName="cleanup"
Nov 03 13:30:33.821827 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.821589    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerName="cleanup"
Nov 03 13:30:33.827199 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.827157    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:33.833952 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.833924    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"]
Nov 03 13:30:33.893397 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.893352    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:33.893579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.893467    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:33.893579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.893564    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2nbh\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-kube-api-access-g2nbh\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:33.893692 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.893673    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:33.996371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.995890    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:33.996371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.995988    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:33.996371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.996099    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2nbh\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-kube-api-access-g2nbh\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:33.996371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.996138    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:34.000362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.997270    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:34.000362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:33.997859    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:34.032408 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:34.028030    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:34.051389 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:34.050685    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2nbh\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-kube-api-access-g2nbh\") pod \"payload-processing-external-model-6958bdd4ff-bhc6f\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:34.143648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:34.143547    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:34.441833 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:34.441803    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"]
Nov 03 13:30:34.442318 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:30:34.442287    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdbee54_b6d3_4ea9_b856_0dc91efeabeb.slice/crio-18f5297e29e7cd7710ecf965383aa774439ccaecedbb8fdb76b4fc2170bab7ac WatchSource:0}: Error finding container 18f5297e29e7cd7710ecf965383aa774439ccaecedbb8fdb76b4fc2170bab7ac: Status 404 returned error can't find the container with id 18f5297e29e7cd7710ecf965383aa774439ccaecedbb8fdb76b4fc2170bab7ac
Nov 03 13:30:34.752662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:34.752565    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f" event={"ID":"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb","Type":"ContainerStarted","Data":"0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe"}
Nov 03 13:30:34.752662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:34.752621    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f" event={"ID":"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb","Type":"ContainerStarted","Data":"18f5297e29e7cd7710ecf965383aa774439ccaecedbb8fdb76b4fc2170bab7ac"}
Nov 03 13:30:34.752662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:34.752647    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:34.772083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:34.772019    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f" podStartSLOduration=1.772004988 podStartE2EDuration="1.772004988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:34.767675602 +0000 UTC m=+1322.579343824" watchObservedRunningTime="2026-11-03 13:30:34.772004988 +0000 UTC m=+1322.583673176"
Nov 03 13:30:45.762526 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:45.762495    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:30:45.815253 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:45.815213    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-877757f87-7znqp"]
Nov 03 13:30:50.815774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:50.815734    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp" podUID="479b6cb0-15f5-47a9-9d06-b19a7a2d895f" containerName="payload-processing" containerID="cri-o://57a33410cd3706822d2388e68a42b8566c473d550dd6ec119fa4f71ea1613bff" gracePeriod=25
Nov 03 13:30:51.041684 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.041643    4447 generic.go:356] "Generic (PLEG): container finished" podID="479b6cb0-15f5-47a9-9d06-b19a7a2d895f" containerID="57a33410cd3706822d2388e68a42b8566c473d550dd6ec119fa4f71ea1613bff" exitCode=0
Nov 03 13:30:51.041862 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.041745    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp" event={"ID":"479b6cb0-15f5-47a9-9d06-b19a7a2d895f","Type":"ContainerDied","Data":"57a33410cd3706822d2388e68a42b8566c473d550dd6ec119fa4f71ea1613bff"}
Nov 03 13:30:51.147863 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.147834    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:30:51.260237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.260203    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-routing-overlay-volume\") pod \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") "
Nov 03 13:30:51.260481 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.260287    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-kube-api-access-bdck7\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-kube-api-access-bdck7\") pod \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") "
Nov 03 13:30:51.260481 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.260367    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-provider-credentials\") pod \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") "
Nov 03 13:30:51.260596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.260568    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-plugins-config-volume\") pod \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\" (UID: \"479b6cb0-15f5-47a9-9d06-b19a7a2d895f\") "
Nov 03 13:30:51.260648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.260584    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-routing-overlay-volume" pod "479b6cb0-15f5-47a9-9d06-b19a7a2d895f" (UID: "479b6cb0-15f5-47a9-9d06-b19a7a2d895f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:51.260953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.260930    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-plugins-config-volume" pod "479b6cb0-15f5-47a9-9d06-b19a7a2d895f" (UID: "479b6cb0-15f5-47a9-9d06-b19a7a2d895f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:51.261194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.261177    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:51.261242 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.261196    4447 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-routing-overlay-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:51.262629 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.262603    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-provider-credentials" pod "479b6cb0-15f5-47a9-9d06-b19a7a2d895f" (UID: "479b6cb0-15f5-47a9-9d06-b19a7a2d895f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:51.262752 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.262729    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-kube-api-access-bdck7" pod "479b6cb0-15f5-47a9-9d06-b19a7a2d895f" (UID: "479b6cb0-15f5-47a9-9d06-b19a7a2d895f"). InnerVolumeSpecName "kube-api-access-bdck7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:51.362934 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.362848    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdck7\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-kube-api-access-bdck7\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:51.362934 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:51.362880    4447 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/479b6cb0-15f5-47a9-9d06-b19a7a2d895f-provider-credentials\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:52.050173 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:52.050136    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp" event={"ID":"479b6cb0-15f5-47a9-9d06-b19a7a2d895f","Type":"ContainerDied","Data":"e5164a1dea2361c33f12926155b4ccd08ae4eadec1e5f1986f87f750c7827ef4"}
Nov 03 13:30:52.050665 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:52.050178    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-877757f87-7znqp"
Nov 03 13:30:52.050665 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:52.050209    4447 scope.go:122] "RemoveContainer" containerID="57a33410cd3706822d2388e68a42b8566c473d550dd6ec119fa4f71ea1613bff"
Nov 03 13:30:52.078152 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:52.078118    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-877757f87-7znqp"]
Nov 03 13:30:52.081821 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:52.081795    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-877757f87-7znqp"]
Nov 03 13:30:52.322671 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:52.322591    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479b6cb0-15f5-47a9-9d06-b19a7a2d895f" path="/var/lib/kubelet/pods/479b6cb0-15f5-47a9-9d06-b19a7a2d895f/volumes"
Nov 03 13:30:53.011203 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.011165    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"]
Nov 03 13:30:53.011969 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.011946    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerName="cleanup"
Nov 03 13:30:53.011969 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.011969    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerName="cleanup"
Nov 03 13:30:53.012154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.011996    4447 cpu_manager.go:414] "containerMap: removing container" podUID="479b6cb0-15f5-47a9-9d06-b19a7a2d895f" containerName="payload-processing"
Nov 03 13:30:53.012154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.012003    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479b6cb0-15f5-47a9-9d06-b19a7a2d895f" containerName="payload-processing"
Nov 03 13:30:53.012154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.012026    4447 cpu_manager.go:414] "containerMap: removing container" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerName="cleanup"
Nov 03 13:30:53.012154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.012034    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c2b332-d2b7-4725-8f15-9df83a2e8322" containerName="cleanup"
Nov 03 13:30:53.012389 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.012192    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72c0e7c-ba47-4329-92cf-034ab65661a3" containerName="cleanup"
Nov 03 13:30:53.012389 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.012208    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfeff074-799d-4342-be32-7d16639582aa" containerName="cleanup"
Nov 03 13:30:53.012389 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.012222    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="479b6cb0-15f5-47a9-9d06-b19a7a2d895f" containerName="payload-processing"
Nov 03 13:30:53.012389 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.012243    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b83265-83ad-4ecb-bce0-631d453db1fe" containerName="cleanup"
Nov 03 13:30:53.017053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.017030    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.028673 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.027398    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-08111b-serving-cert\""
Nov 03 13:30:53.031175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.031151    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"]
Nov 03 13:30:53.090281 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.090241    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nfj\" (UniqueName: \"kubernetes.io/projected/024e3618-244e-4770-980b-1d3b0004341c-kube-api-access-84nfj\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.090694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.090435    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024e3618-244e-4770-980b-1d3b0004341c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.090694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.090465    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-metrics-certs\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.090694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.090569    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.192505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.192469    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024e3618-244e-4770-980b-1d3b0004341c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.192505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.192510    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-metrics-certs\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.192749 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.192562    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.192831 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.192800    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nfj\" (UniqueName: \"kubernetes.io/projected/024e3618-244e-4770-980b-1d3b0004341c-kube-api-access-84nfj\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.193268 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.193247    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024e3618-244e-4770-980b-1d3b0004341c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.195313 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.195291    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.195396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.195291    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-metrics-certs\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.201023 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.201002    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nfj\" (UniqueName: \"kubernetes.io/projected/024e3618-244e-4770-980b-1d3b0004341c-kube-api-access-84nfj\") pod \"maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.341941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.341850    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:53.693679 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.691475    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"]
Nov 03 13:30:53.702327 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:30:53.702291    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024e3618_244e_4770_980b_1d3b0004341c.slice/crio-622dd5d215c970296fdde148dbf1b2bb3cdacf4e101d63d023fa5b64cdad0725 WatchSource:0}: Error finding container 622dd5d215c970296fdde148dbf1b2bb3cdacf4e101d63d023fa5b64cdad0725: Status 404 returned error can't find the container with id 622dd5d215c970296fdde148dbf1b2bb3cdacf4e101d63d023fa5b64cdad0725
Nov 03 13:30:53.704757 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:53.704730    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:30:54.070058 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:54.069955    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8" event={"ID":"024e3618-244e-4770-980b-1d3b0004341c","Type":"ContainerStarted","Data":"622dd5d215c970296fdde148dbf1b2bb3cdacf4e101d63d023fa5b64cdad0725"}
Nov 03 13:30:55.079129 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:55.079089    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8" event={"ID":"024e3618-244e-4770-980b-1d3b0004341c","Type":"ContainerStarted","Data":"b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430"}
Nov 03 13:30:55.079614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:55.079219    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:30:55.097147 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:30:55.097099    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8" podStartSLOduration=2.431649434 podStartE2EDuration="3.09708578s" totalImagesPullingTime="665.436346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:55.094026425 +0000 UTC m=+1342.905694616" watchObservedRunningTime="2026-11-03 13:30:55.09708578 +0000 UTC m=+1342.908753973"
Nov 03 13:31:00.484494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.484451    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"]
Nov 03 13:31:00.489160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.489134    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.492720 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.492695    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-1a18c8-openshift-default-dockercfg-ff4vz\""
Nov 03 13:31:00.505139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.505112    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"]
Nov 03 13:31:00.515772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.515736    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istiod-ca-cert\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.515905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.515798    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-token\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.515950 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.515912    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-data\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.516060 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.516038    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-envoy\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.516151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.516130    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfkmw\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-kube-api-access-gfkmw\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.516286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.516236    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-credential-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.516677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.516337    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-certs\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.516677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.516438    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-podinfo\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.516677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.516488    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.617984 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.617945    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istiod-ca-cert\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.617993    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-token\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618030    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-data\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618079    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-envoy\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618150    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfkmw\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-kube-api-access-gfkmw\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618186    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-credential-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618242    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-certs\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618316    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-podinfo\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618370    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618521    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-data\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618636    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-certs\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618693    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-credential-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618809    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istiod-ca-cert\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.618951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.618845    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.620469 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.620449    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-envoy\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.620713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.620695    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-podinfo\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.627820 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.627795    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-token\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.628051 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.628029    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfkmw\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-kube-api-access-gfkmw\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:00.804530 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:00.804438    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:01.092754 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:01.092665    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:31:01.102381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:01.102145    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"]
Nov 03 13:31:01.141473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:01.141414    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:31:01.141633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:01.141574    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:31:01.141714 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:01.141689    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:31:02.179624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:02.179588    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" event={"ID":"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5","Type":"ContainerStarted","Data":"db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9"}
Nov 03 13:31:02.180055 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:02.179683    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" event={"ID":"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5","Type":"ContainerStarted","Data":"a4c3962203798698257327a847034f80f4c4821317445bb7253c88646b66f32b"}
Nov 03 13:31:02.203065 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:02.203011    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" podStartSLOduration=2.202995379 podStartE2EDuration="2.202995379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:02.199101064 +0000 UTC m=+1350.010769252" watchObservedRunningTime="2026-11-03 13:31:02.202995379 +0000 UTC m=+1350.014663566"
Nov 03 13:31:02.813371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:02.806639    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:02.813371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:02.811596    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 13:31:02.813371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:02.811658    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 13:31:03.420029 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:31:03.419992    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:31:03.805480 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:03.805440    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 13:31:03.805753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:03.805508    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 13:31:04.805121 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:04.805053    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 13:31:04.805728 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:04.805697    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 13:31:05.808587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:05.808553    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 13:31:05.809177 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:05.809143    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 13:31:06.809367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:06.808837    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 13:31:06.809367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:06.808931    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 13:31:07.151120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:07.151018    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:31:07.151766 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:07.151380    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:31:07.806367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:07.805739    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 13:31:07.806367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:07.805805    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 13:31:08.974063 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:08.974028    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:08.975034 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:08.975011    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:08.975906 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:08.975882    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:31:33.526137 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:31:33.526088    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:31:53.369211 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:53.369131    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"]
Nov 03 13:31:53.403835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:53.403797    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"]
Nov 03 13:31:54.476036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:54.475973    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"]
Nov 03 13:31:58.370494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.370425    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f" podUID="8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" containerName="payload-processing" containerID="cri-o://0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe" gracePeriod=25
Nov 03 13:31:58.405154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.405109    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8" podUID="d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" containerName="payload-processing" containerID="cri-o://e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2" gracePeriod=25
Nov 03 13:31:58.694876 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.694845    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:31:58.826417 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.826385    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-routing-overlay-volume\") pod \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") "
Nov 03 13:31:58.826594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.826439    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-plugins-config-volume\") pod \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") "
Nov 03 13:31:58.826594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.826559    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-kube-api-access-s5nsq\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-kube-api-access-s5nsq\") pod \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") "
Nov 03 13:31:58.826594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.826592    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-provider-credentials\") pod \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\" (UID: \"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5\") "
Nov 03 13:31:58.826829 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.826800    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-routing-overlay-volume" pod "d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" (UID: "d93c6a29-128b-4fcf-949c-3c9a78d5dcf5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:58.826873 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.826822    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-plugins-config-volume" pod "d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" (UID: "d93c6a29-128b-4fcf-949c-3c9a78d5dcf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:58.827180 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.827157    4447 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-routing-overlay-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:58.827180 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.827179    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:58.828722 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.828697    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-provider-credentials" pod "d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" (UID: "d93c6a29-128b-4fcf-949c-3c9a78d5dcf5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:58.828807 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.828719    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-kube-api-access-s5nsq" pod "d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" (UID: "d93c6a29-128b-4fcf-949c-3c9a78d5dcf5"). InnerVolumeSpecName "kube-api-access-s5nsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:58.903253 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.903220    4447 generic.go:356] "Generic (PLEG): container finished" podID="d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" containerID="e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2" exitCode=0
Nov 03 13:31:58.903434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.903263    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"
Nov 03 13:31:58.903434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.903306    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8" event={"ID":"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5","Type":"ContainerDied","Data":"e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2"}
Nov 03 13:31:58.903434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.903373    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8" event={"ID":"d93c6a29-128b-4fcf-949c-3c9a78d5dcf5","Type":"ContainerDied","Data":"a254069cd633859826ce4f12aa4a6aea565c5c36db169b7f8cc38a93db1d948b"}
Nov 03 13:31:58.903434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.903403    4447 scope.go:122] "RemoveContainer" containerID="e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2"
Nov 03 13:31:58.916297 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.916181    4447 scope.go:122] "RemoveContainer" containerID="e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2"
Nov 03 13:31:58.918044 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:31:58.918018    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2\": container with ID starting with e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2 not found: ID does not exist" containerID="e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2"
Nov 03 13:31:58.918157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.918049    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2"} err="failed to get container status \"e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2\": rpc error: code = NotFound desc = could not find container \"e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2\": container with ID starting with e8e01670438fb52a558c07ea48aa5cf62b8a7ae72db05b976cb9144aaa4e3dc2 not found: ID does not exist"
Nov 03 13:31:58.928487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.928458    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5nsq\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-kube-api-access-s5nsq\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:58.928487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.928487    4447 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5-provider-credentials\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:58.929735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.929714    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"]
Nov 03 13:31:58.933494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:58.933469    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-f954t2rj8"]
Nov 03 13:31:59.476524 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.476461    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t" podUID="608fc87f-1493-4fa2-9926-1836e87f0e1c" containerName="payload-processing" containerID="cri-o://2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6" gracePeriod=25
Nov 03 13:31:59.780480 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.780454    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:31:59.840136 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.840103    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-kube-api-access-n5rn7\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-kube-api-access-n5rn7\") pod \"608fc87f-1493-4fa2-9926-1836e87f0e1c\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") "
Nov 03 13:31:59.840337 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.840191    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-routing-overlay-volume\") pod \"608fc87f-1493-4fa2-9926-1836e87f0e1c\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") "
Nov 03 13:31:59.840337 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.840227    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-provider-credentials\") pod \"608fc87f-1493-4fa2-9926-1836e87f0e1c\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") "
Nov 03 13:31:59.840510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.840388    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-plugins-config-volume\") pod \"608fc87f-1493-4fa2-9926-1836e87f0e1c\" (UID: \"608fc87f-1493-4fa2-9926-1836e87f0e1c\") "
Nov 03 13:31:59.840908 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.840845    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-routing-overlay-volume" pod "608fc87f-1493-4fa2-9926-1836e87f0e1c" (UID: "608fc87f-1493-4fa2-9926-1836e87f0e1c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:59.841075 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.840990    4447 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-routing-overlay-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:59.841251 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.841227    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-plugins-config-volume" pod "608fc87f-1493-4fa2-9926-1836e87f0e1c" (UID: "608fc87f-1493-4fa2-9926-1836e87f0e1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:59.843210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.843180    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-kube-api-access-n5rn7" pod "608fc87f-1493-4fa2-9926-1836e87f0e1c" (UID: "608fc87f-1493-4fa2-9926-1836e87f0e1c"). InnerVolumeSpecName "kube-api-access-n5rn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:59.843690 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.843653    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-provider-credentials" pod "608fc87f-1493-4fa2-9926-1836e87f0e1c" (UID: "608fc87f-1493-4fa2-9926-1836e87f0e1c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:59.865016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.864990    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:31:59.912196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.912167    4447 generic.go:356] "Generic (PLEG): container finished" podID="608fc87f-1493-4fa2-9926-1836e87f0e1c" containerID="2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6" exitCode=0
Nov 03 13:31:59.912417 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.912228    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"
Nov 03 13:31:59.912417 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.912254    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t" event={"ID":"608fc87f-1493-4fa2-9926-1836e87f0e1c","Type":"ContainerDied","Data":"2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6"}
Nov 03 13:31:59.912417 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.912305    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t" event={"ID":"608fc87f-1493-4fa2-9926-1836e87f0e1c","Type":"ContainerDied","Data":"83f7780fbc8ca6d56384f988ed24a9f7a400feae344be2e48624c3708985d78d"}
Nov 03 13:31:59.912417 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.912331    4447 scope.go:122] "RemoveContainer" containerID="2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6"
Nov 03 13:31:59.916099 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.916053    4447 generic.go:356] "Generic (PLEG): container finished" podID="8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" containerID="0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe" exitCode=0
Nov 03 13:31:59.916229 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.916208    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"
Nov 03 13:31:59.916556 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.916534    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f" event={"ID":"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb","Type":"ContainerDied","Data":"0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe"}
Nov 03 13:31:59.916665 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.916596    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f" event={"ID":"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb","Type":"ContainerDied","Data":"18f5297e29e7cd7710ecf965383aa774439ccaecedbb8fdb76b4fc2170bab7ac"}
Nov 03 13:31:59.924745 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.924665    4447 scope.go:122] "RemoveContainer" containerID="2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6"
Nov 03 13:31:59.924962 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:31:59.924937    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6\": container with ID starting with 2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6 not found: ID does not exist" containerID="2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6"
Nov 03 13:31:59.925046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.924974    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6"} err="failed to get container status \"2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6\": rpc error: code = NotFound desc = could not find container \"2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6\": container with ID starting with 2d89916af96d1ccdec7b129cc0f962840f6d4a91b7045ea198fa0623b3138eb6 not found: ID does not exist"
Nov 03 13:31:59.925046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.924997    4447 scope.go:122] "RemoveContainer" containerID="0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe"
Nov 03 13:31:59.935956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.935940    4447 scope.go:122] "RemoveContainer" containerID="0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe"
Nov 03 13:31:59.936273 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:31:59.936251    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe\": container with ID starting with 0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe not found: ID does not exist" containerID="0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe"
Nov 03 13:31:59.936386 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.936282    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe"} err="failed to get container status \"0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe\": rpc error: code = NotFound desc = could not find container \"0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe\": container with ID starting with 0f5603d1c5c165166a47235a9097cbe0bf110b9e33df324691f87d84d88ee2fe not found: ID does not exist"
Nov 03 13:31:59.941725 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.941706    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-kube-api-access-g2nbh\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-kube-api-access-g2nbh\") pod \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") "
Nov 03 13:31:59.941849 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.941830    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-provider-credentials\") pod \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") "
Nov 03 13:31:59.943370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.942005    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-plugins-config-volume\") pod \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") "
Nov 03 13:31:59.943370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.942063    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-routing-overlay-volume\") pod \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\" (UID: \"8cdbee54-b6d3-4ea9-b856-0dc91efeabeb\") "
Nov 03 13:31:59.943370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.942834    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rn7\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-kube-api-access-n5rn7\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:59.943370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.943313    4447 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608fc87f-1493-4fa2-9926-1836e87f0e1c-provider-credentials\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:59.943370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.943332    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608fc87f-1493-4fa2-9926-1836e87f0e1c-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:59.943684 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.943574    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-routing-overlay-volume" pod "8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" (UID: "8cdbee54-b6d3-4ea9-b856-0dc91efeabeb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:59.944002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.943979    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-plugins-config-volume" pod "8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" (UID: "8cdbee54-b6d3-4ea9-b856-0dc91efeabeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:59.944731 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.944706    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-kube-api-access-g2nbh" pod "8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" (UID: "8cdbee54-b6d3-4ea9-b856-0dc91efeabeb"). InnerVolumeSpecName "kube-api-access-g2nbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:59.945045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.945029    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-provider-credentials" pod "8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" (UID: "8cdbee54-b6d3-4ea9-b856-0dc91efeabeb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:59.950657 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.950637    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"]
Nov 03 13:31:59.956624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:31:59.956602    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5468xx76t"]
Nov 03 13:32:00.046232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:00.046096    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2nbh\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-kube-api-access-g2nbh\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:00.046232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:00.046139    4447 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-provider-credentials\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:00.046232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:00.046156    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:00.046232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:00.046172    4447 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb-routing-overlay-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:00.241027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:00.240994    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"]
Nov 03 13:32:00.251144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:00.251106    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-bhc6f"]
Nov 03 13:32:00.321197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:00.321120    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608fc87f-1493-4fa2-9926-1836e87f0e1c" path="/var/lib/kubelet/pods/608fc87f-1493-4fa2-9926-1836e87f0e1c/volumes"
Nov 03 13:32:00.321791 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:00.321766    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" path="/var/lib/kubelet/pods/8cdbee54-b6d3-4ea9-b856-0dc91efeabeb/volumes"
Nov 03 13:32:00.322228 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:00.322209    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" path="/var/lib/kubelet/pods/d93c6a29-128b-4fcf-949c-3c9a78d5dcf5/volumes"
Nov 03 13:32:03.438852 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:32:03.438808    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:32:03.729457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.728292    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"]
Nov 03 13:32:03.729457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.729380    4447 cpu_manager.go:414] "containerMap: removing container" podUID="8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" containerName="payload-processing"
Nov 03 13:32:03.729457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.729402    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" containerName="payload-processing"
Nov 03 13:32:03.729773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.729463    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" containerName="payload-processing"
Nov 03 13:32:03.729773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.729471    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" containerName="payload-processing"
Nov 03 13:32:03.729773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.729489    4447 cpu_manager.go:414] "containerMap: removing container" podUID="608fc87f-1493-4fa2-9926-1836e87f0e1c" containerName="payload-processing"
Nov 03 13:32:03.729773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.729497    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608fc87f-1493-4fa2-9926-1836e87f0e1c" containerName="payload-processing"
Nov 03 13:32:03.729773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.729658    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="608fc87f-1493-4fa2-9926-1836e87f0e1c" containerName="payload-processing"
Nov 03 13:32:03.729773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.729678    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cdbee54-b6d3-4ea9-b856-0dc91efeabeb" containerName="payload-processing"
Nov 03 13:32:03.729773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.729696    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93c6a29-128b-4fcf-949c-3c9a78d5dcf5" containerName="payload-processing"
Nov 03 13:32:03.743794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.743763    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.748511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.748088    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"]
Nov 03 13:32:03.753796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.753764    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"openshift-service-ca.crt\""
Nov 03 13:32:03.753995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.753888    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-e2e--d832c86e-dockercfg-sq45c\""
Nov 03 13:32:03.753995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.753955    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-de83083e156c\""
Nov 03 13:32:03.754118 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.754042    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"kube-root-ca.crt\""
Nov 03 13:32:03.754118 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.754101    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"routing-overlay\""
Nov 03 13:32:03.754221 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.754152    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1a18c8\""
Nov 03 13:32:03.837447 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.828749    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.837447 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.828892    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.837447 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.829052    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.837447 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.829133    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzp8r\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-kube-api-access-jzp8r\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.970372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.946793    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.970372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.964405    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.970372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.964604    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.970372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.964684    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzp8r\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-kube-api-access-jzp8r\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.970372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.965982    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.970372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.969321    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:03.970372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:03.970065    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:04.002255 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:04.002063    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzp8r\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-kube-api-access-jzp8r\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:04.088320 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:04.088282    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:04.387307 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:04.387276    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"]
Nov 03 13:32:04.388525 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:32:04.388495    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da20532_ab3c_48e8_8891_55bbceb162f5.slice/crio-6c34b6ce9888bf1a10bbc8cc88a18781f669a7056c1e60756d2aed06de1fcad4 WatchSource:0}: Error finding container 6c34b6ce9888bf1a10bbc8cc88a18781f669a7056c1e60756d2aed06de1fcad4: Status 404 returned error can't find the container with id 6c34b6ce9888bf1a10bbc8cc88a18781f669a7056c1e60756d2aed06de1fcad4
Nov 03 13:32:05.001271 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:05.001224    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc" event={"ID":"3da20532-ab3c-48e8-8891-55bbceb162f5","Type":"ContainerStarted","Data":"c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab"}
Nov 03 13:32:05.001739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:05.001311    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc" event={"ID":"3da20532-ab3c-48e8-8891-55bbceb162f5","Type":"ContainerStarted","Data":"6c34b6ce9888bf1a10bbc8cc88a18781f669a7056c1e60756d2aed06de1fcad4"}
Nov 03 13:32:05.001739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:05.001332    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:05.033113 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:05.033047    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc" podStartSLOduration=2.033032812 podStartE2EDuration="2.033032812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:05.028042951 +0000 UTC m=+1412.839711175" watchObservedRunningTime="2026-11-03 13:32:05.033032812 +0000 UTC m=+1412.844701000"
Nov 03 13:32:16.012090 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:16.012060    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:26.018244 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:26.018204    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"]
Nov 03 13:32:31.019150 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.019106    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc" podUID="3da20532-ab3c-48e8-8891-55bbceb162f5" containerName="payload-processing" containerID="cri-o://c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab" gracePeriod=25
Nov 03 13:32:31.306842 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.306815    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:31.381384 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.381328    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-routing-overlay-volume\") pod \"3da20532-ab3c-48e8-8891-55bbceb162f5\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") "
Nov 03 13:32:31.381582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.381476    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-kube-api-access-jzp8r\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-kube-api-access-jzp8r\") pod \"3da20532-ab3c-48e8-8891-55bbceb162f5\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") "
Nov 03 13:32:31.381582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.381542    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-provider-credentials\") pod \"3da20532-ab3c-48e8-8891-55bbceb162f5\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") "
Nov 03 13:32:31.381882 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.381838    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-plugins-config-volume\") pod \"3da20532-ab3c-48e8-8891-55bbceb162f5\" (UID: \"3da20532-ab3c-48e8-8891-55bbceb162f5\") "
Nov 03 13:32:31.383508 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.383480    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-routing-overlay-volume" pod "3da20532-ab3c-48e8-8891-55bbceb162f5" (UID: "3da20532-ab3c-48e8-8891-55bbceb162f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:31.386155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.386128    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-kube-api-access-jzp8r" pod "3da20532-ab3c-48e8-8891-55bbceb162f5" (UID: "3da20532-ab3c-48e8-8891-55bbceb162f5"). InnerVolumeSpecName "kube-api-access-jzp8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:31.386310 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.386230    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-plugins-config-volume" pod "3da20532-ab3c-48e8-8891-55bbceb162f5" (UID: "3da20532-ab3c-48e8-8891-55bbceb162f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:31.392324 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.392284    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-provider-credentials" pod "3da20532-ab3c-48e8-8891-55bbceb162f5" (UID: "3da20532-ab3c-48e8-8891-55bbceb162f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:31.484429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.484390    4447 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-routing-overlay-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:31.484429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.484423    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzp8r\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-kube-api-access-jzp8r\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:31.484429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.484432    4447 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3da20532-ab3c-48e8-8891-55bbceb162f5-provider-credentials\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:31.484682 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.484443    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da20532-ab3c-48e8-8891-55bbceb162f5-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:31.746373 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.741998    4447 generic.go:356] "Generic (PLEG): container finished" podID="3da20532-ab3c-48e8-8891-55bbceb162f5" containerID="c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab" exitCode=0
Nov 03 13:32:31.746791 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.743537    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc" event={"ID":"3da20532-ab3c-48e8-8891-55bbceb162f5","Type":"ContainerDied","Data":"c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab"}
Nov 03 13:32:31.746989 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.746969    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc" event={"ID":"3da20532-ab3c-48e8-8891-55bbceb162f5","Type":"ContainerDied","Data":"6c34b6ce9888bf1a10bbc8cc88a18781f669a7056c1e60756d2aed06de1fcad4"}
Nov 03 13:32:31.747143 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.747122    4447 scope.go:122] "RemoveContainer" containerID="c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab"
Nov 03 13:32:31.747368 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.743626    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"
Nov 03 13:32:31.819842 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.819817    4447 scope.go:122] "RemoveContainer" containerID="c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab"
Nov 03 13:32:31.824605 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:32:31.824480    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab\": container with ID starting with c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab not found: ID does not exist" containerID="c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab"
Nov 03 13:32:31.824605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.824551    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab"} err="failed to get container status \"c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab\": rpc error: code = NotFound desc = could not find container \"c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab\": container with ID starting with c04c0faf5885bc0016fd14981e55a37ac229e84575f0568d546f907d829bc4ab not found: ID does not exist"
Nov 03 13:32:31.850445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.850403    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"]
Nov 03 13:32:31.856277 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:31.856239    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545h9gzc"]
Nov 03 13:32:32.325400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:32.325365    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da20532-ab3c-48e8-8891-55bbceb162f5" path="/var/lib/kubelet/pods/3da20532-ab3c-48e8-8891-55bbceb162f5/volumes"
Nov 03 13:32:33.411890 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:32:33.411855    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:32:48.855327 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.855295    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"]
Nov 03 13:32:48.855937 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.855922    4447 cpu_manager.go:414] "containerMap: removing container" podUID="3da20532-ab3c-48e8-8891-55bbceb162f5" containerName="payload-processing"
Nov 03 13:32:48.855981 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.855938    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da20532-ab3c-48e8-8891-55bbceb162f5" containerName="payload-processing"
Nov 03 13:32:48.856078 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.856068    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da20532-ab3c-48e8-8891-55bbceb162f5" containerName="payload-processing"
Nov 03 13:32:48.862148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.862125    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:48.870648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.870608    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-ec4cc2d7b734\""
Nov 03 13:32:48.870648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.870608    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-c03d3bf9f10f\""
Nov 03 13:32:48.870893 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.870709    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"e2e-cfl-e87c5e-provider-cred\""
Nov 03 13:32:48.870893 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.870822    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"]
Nov 03 13:32:48.871458 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.871435    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"payload-processing-external-model-e2e--f52a173c-dockercfg-d5wkp\""
Nov 03 13:32:48.871786 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.871763    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"routing-overlay\""
Nov 03 13:32:48.871786 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.871779    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"openshift-service-ca.crt\""
Nov 03 13:32:48.871968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.871897    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a62d22\""
Nov 03 13:32:48.872030 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.872014    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"kube-root-ca.crt\""
Nov 03 13:32:48.927040 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.927002    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:48.927220 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.927063    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9jv\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-kube-api-access-nf9jv\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:48.927375 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.927334    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:48.927438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.927390    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:48.968921 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:48.968893    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-4d57ed2d3b27\""
Nov 03 13:32:49.028373 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.028326    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:49.028577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.028385    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:49.028577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.028465    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:49.028577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.028508    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9jv\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-kube-api-access-nf9jv\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:49.029032 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.029004    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:49.029280 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.029261    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:49.031135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.031113    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:49.041779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.041689    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9jv\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-kube-api-access-nf9jv\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:49.184137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.184101    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:49.469974 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:49.469943    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"]
Nov 03 13:32:49.471237 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:32:49.471211    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c794c4_b377_488d_8042_4efb90f80071.slice/crio-576b48024f941f465ab5d9c35668d4fcacc4ef8d6225502320259582dd1f445d WatchSource:0}: Error finding container 576b48024f941f465ab5d9c35668d4fcacc4ef8d6225502320259582dd1f445d: Status 404 returned error can't find the container with id 576b48024f941f465ab5d9c35668d4fcacc4ef8d6225502320259582dd1f445d
Nov 03 13:32:50.108777 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:50.108737    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4" event={"ID":"d6c794c4-b377-488d-8042-4efb90f80071","Type":"ContainerStarted","Data":"bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5"}
Nov 03 13:32:50.109263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:50.108800    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4" event={"ID":"d6c794c4-b377-488d-8042-4efb90f80071","Type":"ContainerStarted","Data":"576b48024f941f465ab5d9c35668d4fcacc4ef8d6225502320259582dd1f445d"}
Nov 03 13:32:50.109263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:50.108913    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:32:50.126614 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:50.126571    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4" podStartSLOduration=2.126558674 podStartE2EDuration="2.126558674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:50.124817519 +0000 UTC m=+1457.936485728" watchObservedRunningTime="2026-11-03 13:32:50.126558674 +0000 UTC m=+1457.938226862"
Nov 03 13:32:51.159891 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.159849    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"]
Nov 03 13:32:51.178618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.178579    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"]
Nov 03 13:32:51.178802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.178712    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.185446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.185298    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dd2bbf-serving-cert\""
Nov 03 13:32:51.286096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.286049    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0ccbe3-8de9-4c60-9448-67dab863b805-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.286096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.286106    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fkrn\" (UniqueName: \"kubernetes.io/projected/aa0ccbe3-8de9-4c60-9448-67dab863b805-kube-api-access-4fkrn\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.286313 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.286278    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-metrics-certs\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.286510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.286490    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-maas-api-tls\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.387764 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.387725    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fkrn\" (UniqueName: \"kubernetes.io/projected/aa0ccbe3-8de9-4c60-9448-67dab863b805-kube-api-access-4fkrn\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.388032 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.387858    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-metrics-certs\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.388032 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.388007    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-maas-api-tls\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.388307 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.388284    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0ccbe3-8de9-4c60-9448-67dab863b805-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.389393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.389138    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0ccbe3-8de9-4c60-9448-67dab863b805-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.399473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.399433    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-metrics-certs\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.400197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.400170    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-maas-api-tls\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.402884 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.402743    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fkrn\" (UniqueName: \"kubernetes.io/projected/aa0ccbe3-8de9-4c60-9448-67dab863b805-kube-api-access-4fkrn\") pod \"maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.500251 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.500215    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:51.830621 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:51.820867    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"]
Nov 03 13:32:51.854606 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:32:51.838396    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0ccbe3_8de9_4c60_9448_67dab863b805.slice/crio-2c125cf5f3fe96509a3f9379d6c49e02dd43977e568a6fa00127b67fe20fb137 WatchSource:0}: Error finding container 2c125cf5f3fe96509a3f9379d6c49e02dd43977e568a6fa00127b67fe20fb137: Status 404 returned error can't find the container with id 2c125cf5f3fe96509a3f9379d6c49e02dd43977e568a6fa00127b67fe20fb137
Nov 03 13:32:52.186926 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:52.186890    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6" event={"ID":"aa0ccbe3-8de9-4c60-9448-67dab863b805","Type":"ContainerStarted","Data":"2c125cf5f3fe96509a3f9379d6c49e02dd43977e568a6fa00127b67fe20fb137"}
Nov 03 13:32:53.196338 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:53.196295    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6" event={"ID":"aa0ccbe3-8de9-4c60-9448-67dab863b805","Type":"ContainerStarted","Data":"194d4ca76d0059c00ff7e5df102acda640c27906e618e974bb40d63e6981d251"}
Nov 03 13:32:53.196839 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:53.196439    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:32:53.217570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:53.217521    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6" podStartSLOduration=1.6997423409999999 podStartE2EDuration="2.217506944s" totalImagesPullingTime="517.764603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:53.213026826 +0000 UTC m=+1461.024695016" watchObservedRunningTime="2026-11-03 13:32:53.217506944 +0000 UTC m=+1461.029175168"
Nov 03 13:32:59.211927 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:32:59.211896    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:33:00.214979 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.214945    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"]
Nov 03 13:33:00.219752 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.219732    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.224958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.224935    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1ec4b892-serving-cert\""
Nov 03 13:33:00.251987 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.251939    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"]
Nov 03 13:33:00.262981 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.262947    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-maas-api-tls\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.263134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.262992    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnnb\" (UniqueName: \"kubernetes.io/projected/dd130566-cad8-41be-8096-66553b8420ac-kube-api-access-6rnnb\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.263134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.263101    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-metrics-certs\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.263252 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.263186    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd130566-cad8-41be-8096-66553b8420ac-openshift-service-ca\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.364518 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.364486    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd130566-cad8-41be-8096-66553b8420ac-openshift-service-ca\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.364681 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.364628    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-maas-api-tls\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.364681 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.364649    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnnb\" (UniqueName: \"kubernetes.io/projected/dd130566-cad8-41be-8096-66553b8420ac-kube-api-access-6rnnb\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.364760 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.364720    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-metrics-certs\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.365229 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.365203    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd130566-cad8-41be-8096-66553b8420ac-openshift-service-ca\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.367170 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.367153    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-metrics-certs\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.367255 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.367157    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-maas-api-tls\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.374199 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.374180    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnnb\" (UniqueName: \"kubernetes.io/projected/dd130566-cad8-41be-8096-66553b8420ac-kube-api-access-6rnnb\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.555373 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.555274    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:00.922769 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:00.922735    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"]
Nov 03 13:33:01.189372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:01.189236    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:33:01.294867 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:01.292806    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc" event={"ID":"dd130566-cad8-41be-8096-66553b8420ac","Type":"ContainerStarted","Data":"fbe063e72dbb32fbf9aaf4113ad20ce5868164a195f3e3abc2edd0deb7cbc7bb"}
Nov 03 13:33:02.303630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:02.303588    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc" event={"ID":"dd130566-cad8-41be-8096-66553b8420ac","Type":"ContainerStarted","Data":"137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d"}
Nov 03 13:33:02.304048 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:02.303668    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:02.327854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:02.327799    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc" podStartSLOduration=1.7973050640000001 podStartE2EDuration="2.327784021s" totalImagesPullingTime="530.478957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:02.323453015 +0000 UTC m=+1470.135121230" watchObservedRunningTime="2026-11-03 13:33:02.327784021 +0000 UTC m=+1470.139452209"
Nov 03 13:33:03.500368 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:33:03.490825    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:33:06.030303 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:06.018466    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:33:06.030303 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:06.018558    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:33:08.346083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:08.346049    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:33:33.422764 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:33:33.422672    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:33:52.754589 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:52.754539    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"]
Nov 03 13:33:52.755740 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:52.755687    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6" podUID="aa0ccbe3-8de9-4c60-9448-67dab863b805" containerName="maas-api" containerID="cri-o://194d4ca76d0059c00ff7e5df102acda640c27906e618e974bb40d63e6981d251" gracePeriod=30
Nov 03 13:33:52.923027 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:52.922994    4447 generic.go:356] "Generic (PLEG): container finished" podID="aa0ccbe3-8de9-4c60-9448-67dab863b805" containerID="194d4ca76d0059c00ff7e5df102acda640c27906e618e974bb40d63e6981d251" exitCode=0
Nov 03 13:33:52.923205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:52.923127    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6" event={"ID":"aa0ccbe3-8de9-4c60-9448-67dab863b805","Type":"ContainerDied","Data":"194d4ca76d0059c00ff7e5df102acda640c27906e618e974bb40d63e6981d251"}
Nov 03 13:33:53.269013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.268984    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:33:53.356453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.356412    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-metrics-certs\") pod \"aa0ccbe3-8de9-4c60-9448-67dab863b805\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") "
Nov 03 13:33:53.356661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.356511    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-maas-api-tls\") pod \"aa0ccbe3-8de9-4c60-9448-67dab863b805\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") "
Nov 03 13:33:53.356661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.356564    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0ccbe3-8de9-4c60-9448-67dab863b805-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0ccbe3-8de9-4c60-9448-67dab863b805-openshift-service-ca\") pod \"aa0ccbe3-8de9-4c60-9448-67dab863b805\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") "
Nov 03 13:33:53.356798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.356676    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0ccbe3-8de9-4c60-9448-67dab863b805-kube-api-access-4fkrn\" (UniqueName: \"kubernetes.io/projected/aa0ccbe3-8de9-4c60-9448-67dab863b805-kube-api-access-4fkrn\") pod \"aa0ccbe3-8de9-4c60-9448-67dab863b805\" (UID: \"aa0ccbe3-8de9-4c60-9448-67dab863b805\") "
Nov 03 13:33:53.357039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.357009    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0ccbe3-8de9-4c60-9448-67dab863b805-openshift-service-ca" pod "aa0ccbe3-8de9-4c60-9448-67dab863b805" (UID: "aa0ccbe3-8de9-4c60-9448-67dab863b805"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:33:53.357959 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.357932    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0ccbe3-8de9-4c60-9448-67dab863b805-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:53.359043 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.359021    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-maas-api-tls" pod "aa0ccbe3-8de9-4c60-9448-67dab863b805" (UID: "aa0ccbe3-8de9-4c60-9448-67dab863b805"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:33:53.359172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.359156    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-metrics-certs" pod "aa0ccbe3-8de9-4c60-9448-67dab863b805" (UID: "aa0ccbe3-8de9-4c60-9448-67dab863b805"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:33:53.359859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.359842    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0ccbe3-8de9-4c60-9448-67dab863b805-kube-api-access-4fkrn" pod "aa0ccbe3-8de9-4c60-9448-67dab863b805" (UID: "aa0ccbe3-8de9-4c60-9448-67dab863b805"). InnerVolumeSpecName "kube-api-access-4fkrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:33:53.459881 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.459847    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:53.459881 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.459879    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0ccbe3-8de9-4c60-9448-67dab863b805-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:53.460172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.459889    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fkrn\" (UniqueName: \"kubernetes.io/projected/aa0ccbe3-8de9-4c60-9448-67dab863b805-kube-api-access-4fkrn\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:53.934928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.934890    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6" event={"ID":"aa0ccbe3-8de9-4c60-9448-67dab863b805","Type":"ContainerDied","Data":"2c125cf5f3fe96509a3f9379d6c49e02dd43977e568a6fa00127b67fe20fb137"}
Nov 03 13:33:53.935621 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.934926    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"
Nov 03 13:33:53.935918 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.935606    4447 scope.go:122] "RemoveContainer" containerID="194d4ca76d0059c00ff7e5df102acda640c27906e618e974bb40d63e6981d251"
Nov 03 13:33:53.972305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.972274    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"]
Nov 03 13:33:53.979217 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:53.979189    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd2bbf-75ffc9f86c-7j5m6"]
Nov 03 13:33:54.322505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:33:54.322423    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0ccbe3-8de9-4c60-9448-67dab863b805" path="/var/lib/kubelet/pods/aa0ccbe3-8de9-4c60-9448-67dab863b805/volumes"
Nov 03 13:34:03.412521 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:03.412484    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:34:03.458400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:03.458363    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"]
Nov 03 13:34:03.458600 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:03.458568    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy" containerID="cri-o://e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a" gracePeriod=30
Nov 03 13:34:07.494098 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:07.489699    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"]
Nov 03 13:34:07.494098 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:07.489946    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy" containerID="cri-o://0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d" gracePeriod=30
Nov 03 13:34:08.735934 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.735858    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:34:08.839455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839423    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-certs\") pod \"db6c06c3-3ebc-413f-b9f2-3295277b123c\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") "
Nov 03 13:34:08.839651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839504    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-kube-api-access-m29mv\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-kube-api-access-m29mv\") pod \"db6c06c3-3ebc-413f-b9f2-3295277b123c\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") "
Nov 03 13:34:08.839651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839550    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-data\") pod \"db6c06c3-3ebc-413f-b9f2-3295277b123c\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") "
Nov 03 13:34:08.839651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839577    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-socket\") pod \"db6c06c3-3ebc-413f-b9f2-3295277b123c\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") "
Nov 03 13:34:08.839821 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839718    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-podinfo\") pod \"db6c06c3-3ebc-413f-b9f2-3295277b123c\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") "
Nov 03 13:34:08.839821 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839758    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-certs" pod "db6c06c3-3ebc-413f-b9f2-3295277b123c" (UID: "db6c06c3-3ebc-413f-b9f2-3295277b123c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:08.839821 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839786    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-envoy\") pod \"db6c06c3-3ebc-413f-b9f2-3295277b123c\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") "
Nov 03 13:34:08.839986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839850    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-data" pod "db6c06c3-3ebc-413f-b9f2-3295277b123c" (UID: "db6c06c3-3ebc-413f-b9f2-3295277b123c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:08.839986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839931    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-credential-socket\") pod \"db6c06c3-3ebc-413f-b9f2-3295277b123c\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") "
Nov 03 13:34:08.839986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839938    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-socket" pod "db6c06c3-3ebc-413f-b9f2-3295277b123c" (UID: "db6c06c3-3ebc-413f-b9f2-3295277b123c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:08.839986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.839984    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-token\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-token\") pod \"db6c06c3-3ebc-413f-b9f2-3295277b123c\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") "
Nov 03 13:34:08.840195 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.840131    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-credential-socket" pod "db6c06c3-3ebc-413f-b9f2-3295277b123c" (UID: "db6c06c3-3ebc-413f-b9f2-3295277b123c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:08.840195 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.840161    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db6c06c3-3ebc-413f-b9f2-3295277b123c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6c06c3-3ebc-413f-b9f2-3295277b123c-istiod-ca-cert\") pod \"db6c06c3-3ebc-413f-b9f2-3295277b123c\" (UID: \"db6c06c3-3ebc-413f-b9f2-3295277b123c\") "
Nov 03 13:34:08.840835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.840505    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6c06c3-3ebc-413f-b9f2-3295277b123c-istiod-ca-cert" pod "db6c06c3-3ebc-413f-b9f2-3295277b123c" (UID: "db6c06c3-3ebc-413f-b9f2-3295277b123c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:08.840835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.840777    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:08.840835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.840792    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:08.840835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.840801    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:08.840835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.840810    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:08.840835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.840818    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6c06c3-3ebc-413f-b9f2-3295277b123c-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:08.842024 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.841996    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-podinfo" pod "db6c06c3-3ebc-413f-b9f2-3295277b123c" (UID: "db6c06c3-3ebc-413f-b9f2-3295277b123c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:08.842136 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.842007    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-envoy" pod "db6c06c3-3ebc-413f-b9f2-3295277b123c" (UID: "db6c06c3-3ebc-413f-b9f2-3295277b123c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:08.842266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.842243    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-kube-api-access-m29mv" pod "db6c06c3-3ebc-413f-b9f2-3295277b123c" (UID: "db6c06c3-3ebc-413f-b9f2-3295277b123c"). InnerVolumeSpecName "kube-api-access-m29mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:08.842464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.842443    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-token" pod "db6c06c3-3ebc-413f-b9f2-3295277b123c" (UID: "db6c06c3-3ebc-413f-b9f2-3295277b123c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:08.942065 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.942025    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:08.942065 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.942058    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:08.942065 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.942068    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:08.942306 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:08.942079    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m29mv\" (UniqueName: \"kubernetes.io/projected/db6c06c3-3ebc-413f-b9f2-3295277b123c-kube-api-access-m29mv\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:09.054835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:09.054732    4447 generic.go:356] "Generic (PLEG): container finished" podID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerID="e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a" exitCode=0
Nov 03 13:34:09.054835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:09.054776    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"
Nov 03 13:34:09.054835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:09.054808    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" event={"ID":"db6c06c3-3ebc-413f-b9f2-3295277b123c","Type":"ContainerDied","Data":"e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a"}
Nov 03 13:34:09.055142 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:09.054855    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc" event={"ID":"db6c06c3-3ebc-413f-b9f2-3295277b123c","Type":"ContainerDied","Data":"2296e475a5f782808aa4ae697d692984713eb774a1917f699f3a6c5fb165204d"}
Nov 03 13:34:09.055142 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:09.054882    4447 scope.go:122] "RemoveContainer" containerID="e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a"
Nov 03 13:34:09.067077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:09.067054    4447 scope.go:122] "RemoveContainer" containerID="e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a"
Nov 03 13:34:09.067444 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:09.067421    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a\": container with ID starting with e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a not found: ID does not exist" containerID="e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a"
Nov 03 13:34:09.067552 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:09.067452    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a"} err="failed to get container status \"e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a\": rpc error: code = NotFound desc = could not find container \"e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a\": container with ID starting with e4e902b9fe22a7d6e43334209d101570fafac2e9287bd4a42fd787d2632c3c0a not found: ID does not exist"
Nov 03 13:34:09.086578 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:09.086551    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"]
Nov 03 13:34:09.093004 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:09.092977    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-b4qlc"]
Nov 03 13:34:10.322162 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:10.322129    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" path="/var/lib/kubelet/pods/db6c06c3-3ebc-413f-b9f2-3295277b123c/volumes"
Nov 03 13:34:11.017854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:11.017811    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"]
Nov 03 13:34:11.018094 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:11.018070    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerName="istio-proxy" containerID="cri-o://ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9" gracePeriod=30
Nov 03 13:34:12.768896 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.768632    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:34:12.912137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.909033    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-credential-socket\") pod \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") "
Nov 03 13:34:12.912481 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.910500    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-credential-socket" pod "4036ac2c-02b8-4eed-9efa-59b749a95b8b" (UID: "4036ac2c-02b8-4eed-9efa-59b749a95b8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:12.912721 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.912698    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-certs\") pod \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") "
Nov 03 13:34:12.913287 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.913147    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-certs" pod "4036ac2c-02b8-4eed-9efa-59b749a95b8b" (UID: "4036ac2c-02b8-4eed-9efa-59b749a95b8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:12.920716 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.914513    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-token\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-token\") pod \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") "
Nov 03 13:34:12.920716 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.915013    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-socket\") pod \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") "
Nov 03 13:34:12.922833 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.922795    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-podinfo\") pod \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") "
Nov 03 13:34:12.922951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.922922    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istiod-ca-cert\") pod \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") "
Nov 03 13:34:12.922951 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.922943    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-kube-api-access-tvkqk\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-kube-api-access-tvkqk\") pod \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") "
Nov 03 13:34:12.923075 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.922999    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-envoy\") pod \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") "
Nov 03 13:34:12.923075 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.923058    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-data\") pod \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\" (UID: \"4036ac2c-02b8-4eed-9efa-59b749a95b8b\") "
Nov 03 13:34:12.923176 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.923081    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-socket" pod "4036ac2c-02b8-4eed-9efa-59b749a95b8b" (UID: "4036ac2c-02b8-4eed-9efa-59b749a95b8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:12.923499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.923452    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istiod-ca-cert" pod "4036ac2c-02b8-4eed-9efa-59b749a95b8b" (UID: "4036ac2c-02b8-4eed-9efa-59b749a95b8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:12.923577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.923546    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-token" pod "4036ac2c-02b8-4eed-9efa-59b749a95b8b" (UID: "4036ac2c-02b8-4eed-9efa-59b749a95b8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:12.923902 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.923878    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:12.923988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.923907    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:12.923988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.923921    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:12.923988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.923935    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:12.923988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.923949    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:12.924579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.924553    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-data" pod "4036ac2c-02b8-4eed-9efa-59b749a95b8b" (UID: "4036ac2c-02b8-4eed-9efa-59b749a95b8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:12.926218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.926154    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-kube-api-access-tvkqk" pod "4036ac2c-02b8-4eed-9efa-59b749a95b8b" (UID: "4036ac2c-02b8-4eed-9efa-59b749a95b8b"). InnerVolumeSpecName "kube-api-access-tvkqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:12.926680 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.926605    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-podinfo" pod "4036ac2c-02b8-4eed-9efa-59b749a95b8b" (UID: "4036ac2c-02b8-4eed-9efa-59b749a95b8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:12.929787 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:12.929139    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-envoy" pod "4036ac2c-02b8-4eed-9efa-59b749a95b8b" (UID: "4036ac2c-02b8-4eed-9efa-59b749a95b8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:13.024942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.024908    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:13.024942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.024939    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvkqk\" (UniqueName: \"kubernetes.io/projected/4036ac2c-02b8-4eed-9efa-59b749a95b8b-kube-api-access-tvkqk\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:13.024942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.024950    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:13.025187 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.024959    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4036ac2c-02b8-4eed-9efa-59b749a95b8b-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:13.092874 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.092841    4447 generic.go:356] "Generic (PLEG): container finished" podID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerID="0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d" exitCode=0
Nov 03 13:34:13.093050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.092892    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"
Nov 03 13:34:13.093050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.092919    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" event={"ID":"4036ac2c-02b8-4eed-9efa-59b749a95b8b","Type":"ContainerDied","Data":"0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d"}
Nov 03 13:34:13.093050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.092982    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n" event={"ID":"4036ac2c-02b8-4eed-9efa-59b749a95b8b","Type":"ContainerDied","Data":"c08175666ee091089580760aa7c3a41b128b47934749c6c08db873c8bb7e4f61"}
Nov 03 13:34:13.093050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.093012    4447 scope.go:122] "RemoveContainer" containerID="0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d"
Nov 03 13:34:13.107569 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.107544    4447 scope.go:122] "RemoveContainer" containerID="0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d"
Nov 03 13:34:13.107895 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:13.107873    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d\": container with ID starting with 0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d not found: ID does not exist" containerID="0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d"
Nov 03 13:34:13.107993 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.107915    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d"} err="failed to get container status \"0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d\": rpc error: code = NotFound desc = could not find container \"0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d\": container with ID starting with 0394dafd2ecad3436a3486288345a38d4caecc940b7f935c2f2c27c514c4cc5d not found: ID does not exist"
Nov 03 13:34:13.120713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.120687    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"]
Nov 03 13:34:13.127653 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:13.127627    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-82w8n"]
Nov 03 13:34:14.321016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:14.320983    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" path="/var/lib/kubelet/pods/4036ac2c-02b8-4eed-9efa-59b749a95b8b/volumes"
Nov 03 13:34:16.350596 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.350558    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:34:16.479974 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.479938    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-credential-socket\") pod \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") "
Nov 03 13:34:16.480156 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.479989    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istiod-ca-cert\") pod \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") "
Nov 03 13:34:16.480156 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480062    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-socket\") pod \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") "
Nov 03 13:34:16.480156 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480143    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-certs\") pod \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") "
Nov 03 13:34:16.480334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480189    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-envoy\") pod \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") "
Nov 03 13:34:16.480334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480255    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-token\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-token\") pod \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") "
Nov 03 13:34:16.480334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480294    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-kube-api-access-k4kzr\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-kube-api-access-k4kzr\") pod \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") "
Nov 03 13:34:16.480334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480305    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-credential-socket" pod "484ecf48-e4c0-4175-b9b3-7a03e31bc001" (UID: "484ecf48-e4c0-4175-b9b3-7a03e31bc001"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.480334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480327    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-data\") pod \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") "
Nov 03 13:34:16.480638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480406    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-podinfo\") pod \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\" (UID: \"484ecf48-e4c0-4175-b9b3-7a03e31bc001\") "
Nov 03 13:34:16.480638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480583    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-certs" pod "484ecf48-e4c0-4175-b9b3-7a03e31bc001" (UID: "484ecf48-e4c0-4175-b9b3-7a03e31bc001"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.480638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480601    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-socket" pod "484ecf48-e4c0-4175-b9b3-7a03e31bc001" (UID: "484ecf48-e4c0-4175-b9b3-7a03e31bc001"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.480638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480620    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istiod-ca-cert" pod "484ecf48-e4c0-4175-b9b3-7a03e31bc001" (UID: "484ecf48-e4c0-4175-b9b3-7a03e31bc001"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:16.480858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.480796    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-data" pod "484ecf48-e4c0-4175-b9b3-7a03e31bc001" (UID: "484ecf48-e4c0-4175-b9b3-7a03e31bc001"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.481311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.481286    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.481448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.481316    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.481448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.481332    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.481448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.481387    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.481448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.481401    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.482748 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.482721    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-kube-api-access-k4kzr" pod "484ecf48-e4c0-4175-b9b3-7a03e31bc001" (UID: "484ecf48-e4c0-4175-b9b3-7a03e31bc001"). InnerVolumeSpecName "kube-api-access-k4kzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:16.482830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.482766    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-token" pod "484ecf48-e4c0-4175-b9b3-7a03e31bc001" (UID: "484ecf48-e4c0-4175-b9b3-7a03e31bc001"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:16.483317 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.483302    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-envoy" pod "484ecf48-e4c0-4175-b9b3-7a03e31bc001" (UID: "484ecf48-e4c0-4175-b9b3-7a03e31bc001"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.483526 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.483497    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-podinfo" pod "484ecf48-e4c0-4175-b9b3-7a03e31bc001" (UID: "484ecf48-e4c0-4175-b9b3-7a03e31bc001"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:16.582505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.582413    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.582505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.582445    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.582505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.582456    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4kzr\" (UniqueName: \"kubernetes.io/projected/484ecf48-e4c0-4175-b9b3-7a03e31bc001-kube-api-access-k4kzr\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.582505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:16.582466    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484ecf48-e4c0-4175-b9b3-7a03e31bc001-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:17.126860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:17.126827    4447 generic.go:356] "Generic (PLEG): container finished" podID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerID="ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9" exitCode=0
Nov 03 13:34:17.127079 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:17.126884    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"
Nov 03 13:34:17.127079 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:17.126908    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" event={"ID":"484ecf48-e4c0-4175-b9b3-7a03e31bc001","Type":"ContainerDied","Data":"ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9"}
Nov 03 13:34:17.127079 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:17.126964    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p" event={"ID":"484ecf48-e4c0-4175-b9b3-7a03e31bc001","Type":"ContainerDied","Data":"e8701dbbb1f6cdb53b2f9731e94cdc2ffbe4e5f2a82ab77b5cc692e6f3f0fd5b"}
Nov 03 13:34:17.127079 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:17.126985    4447 scope.go:122] "RemoveContainer" containerID="ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9"
Nov 03 13:34:17.138083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:17.137910    4447 scope.go:122] "RemoveContainer" containerID="ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9"
Nov 03 13:34:17.138218 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:17.138197    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9\": container with ID starting with ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9 not found: ID does not exist" containerID="ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9"
Nov 03 13:34:17.138283 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:17.138232    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9"} err="failed to get container status \"ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9\": rpc error: code = NotFound desc = could not find container \"ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9\": container with ID starting with ee9318a9029033a4eef72fe564f6d8bd7eb4e080d1d36b6214c445d297be0ec9 not found: ID does not exist"
Nov 03 13:34:17.152418 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:17.152388    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"]
Nov 03 13:34:17.167386 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:17.167332    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-pzj9p"]
Nov 03 13:34:18.212630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:18.212587    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"]
Nov 03 13:34:18.215466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:18.215434    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy" containerID="cri-o://cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68" gracePeriod=30
Nov 03 13:34:18.223961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:18.223924    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"]
Nov 03 13:34:18.224831 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:18.224484    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy" containerID="cri-o://16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f" gracePeriod=30
Nov 03 13:34:18.324336 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:18.324297    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" path="/var/lib/kubelet/pods/484ecf48-e4c0-4175-b9b3-7a03e31bc001/volumes"
Nov 03 13:34:18.568296 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:18.568204    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"]
Nov 03 13:34:18.568547 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:18.568513    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy" containerID="cri-o://e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a" gracePeriod=30
Nov 03 13:34:18.574531 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:18.573517    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"]
Nov 03 13:34:18.574531 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:18.573763    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy" containerID="cri-o://71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf" gracePeriod=30
Nov 03 13:34:23.500318 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.500292    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:34:23.600615 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.600587    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-socket\") pod \"b6043f9d-679a-490b-8753-b32e618cce8c\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") "
Nov 03 13:34:23.600750 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.600729    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-credential-socket\") pod \"b6043f9d-679a-490b-8753-b32e618cce8c\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") "
Nov 03 13:34:23.600808 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.600782    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-certs\") pod \"b6043f9d-679a-490b-8753-b32e618cce8c\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") "
Nov 03 13:34:23.600860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.600838    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6043f9d-679a-490b-8753-b32e618cce8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6043f9d-679a-490b-8753-b32e618cce8c-istio-podinfo\") pod \"b6043f9d-679a-490b-8753-b32e618cce8c\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") "
Nov 03 13:34:23.600860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.600845    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-socket" pod "b6043f9d-679a-490b-8753-b32e618cce8c" (UID: "b6043f9d-679a-490b-8753-b32e618cce8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.600968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.600879    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-envoy\") pod \"b6043f9d-679a-490b-8753-b32e618cce8c\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") "
Nov 03 13:34:23.600968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.600940    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-data\") pod \"b6043f9d-679a-490b-8753-b32e618cce8c\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") "
Nov 03 13:34:23.601074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.600995    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-istio-token\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-istio-token\") pod \"b6043f9d-679a-490b-8753-b32e618cce8c\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") "
Nov 03 13:34:23.601074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601017    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-credential-socket" pod "b6043f9d-679a-490b-8753-b32e618cce8c" (UID: "b6043f9d-679a-490b-8753-b32e618cce8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.601074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601055    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-certs" pod "b6043f9d-679a-490b-8753-b32e618cce8c" (UID: "b6043f9d-679a-490b-8753-b32e618cce8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.601323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601098    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6043f9d-679a-490b-8753-b32e618cce8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6043f9d-679a-490b-8753-b32e618cce8c-istiod-ca-cert\") pod \"b6043f9d-679a-490b-8753-b32e618cce8c\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") "
Nov 03 13:34:23.601323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601200    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-kube-api-access-t7wnl\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-kube-api-access-t7wnl\") pod \"b6043f9d-679a-490b-8753-b32e618cce8c\" (UID: \"b6043f9d-679a-490b-8753-b32e618cce8c\") "
Nov 03 13:34:23.601323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601247    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-data" pod "b6043f9d-679a-490b-8753-b32e618cce8c" (UID: "b6043f9d-679a-490b-8753-b32e618cce8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.601680 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601657    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6043f9d-679a-490b-8753-b32e618cce8c-istiod-ca-cert" pod "b6043f9d-679a-490b-8753-b32e618cce8c" (UID: "b6043f9d-679a-490b-8753-b32e618cce8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:23.601896 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601868    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.601954 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601899    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.601954 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601916    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.601954 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601934    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6043f9d-679a-490b-8753-b32e618cce8c-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.601954 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.601947    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.603457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.603429    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-istio-token" pod "b6043f9d-679a-490b-8753-b32e618cce8c" (UID: "b6043f9d-679a-490b-8753-b32e618cce8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:23.603671 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.603639    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6043f9d-679a-490b-8753-b32e618cce8c-istio-podinfo" pod "b6043f9d-679a-490b-8753-b32e618cce8c" (UID: "b6043f9d-679a-490b-8753-b32e618cce8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:23.604088 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.604070    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-envoy" pod "b6043f9d-679a-490b-8753-b32e618cce8c" (UID: "b6043f9d-679a-490b-8753-b32e618cce8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.604163 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.604103    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-kube-api-access-t7wnl" pod "b6043f9d-679a-490b-8753-b32e618cce8c" (UID: "b6043f9d-679a-490b-8753-b32e618cce8c"). InnerVolumeSpecName "kube-api-access-t7wnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:23.703657 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.703621    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wnl\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-kube-api-access-t7wnl\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.703657 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.703658    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6043f9d-679a-490b-8753-b32e618cce8c-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.703857 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.703677    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6043f9d-679a-490b-8753-b32e618cce8c-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.703857 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.703690    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6043f9d-679a-490b-8753-b32e618cce8c-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.847936 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.847866    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:34:23.910458 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910428    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-token\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-token\") pod \"57c8ebc5-ed58-43e0-96ab-70c7da119904\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") "
Nov 03 13:34:23.910638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910478    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c8ebc5-ed58-43e0-96ab-70c7da119904-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c8ebc5-ed58-43e0-96ab-70c7da119904-istiod-ca-cert\") pod \"57c8ebc5-ed58-43e0-96ab-70c7da119904\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") "
Nov 03 13:34:23.910638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910509    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-socket\") pod \"57c8ebc5-ed58-43e0-96ab-70c7da119904\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") "
Nov 03 13:34:23.910638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910529    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-kube-api-access-jpzzw\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-kube-api-access-jpzzw\") pod \"57c8ebc5-ed58-43e0-96ab-70c7da119904\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") "
Nov 03 13:34:23.910826 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910681    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-podinfo\") pod \"57c8ebc5-ed58-43e0-96ab-70c7da119904\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") "
Nov 03 13:34:23.910826 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910729    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-credential-socket\") pod \"57c8ebc5-ed58-43e0-96ab-70c7da119904\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") "
Nov 03 13:34:23.910826 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910787    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-envoy\") pod \"57c8ebc5-ed58-43e0-96ab-70c7da119904\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") "
Nov 03 13:34:23.910976 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910863    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-data\") pod \"57c8ebc5-ed58-43e0-96ab-70c7da119904\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") "
Nov 03 13:34:23.910976 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910888    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c8ebc5-ed58-43e0-96ab-70c7da119904-istiod-ca-cert" pod "57c8ebc5-ed58-43e0-96ab-70c7da119904" (UID: "57c8ebc5-ed58-43e0-96ab-70c7da119904"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:23.910976 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.910896    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-certs\") pod \"57c8ebc5-ed58-43e0-96ab-70c7da119904\" (UID: \"57c8ebc5-ed58-43e0-96ab-70c7da119904\") "
Nov 03 13:34:23.911135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.911108    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-certs" pod "57c8ebc5-ed58-43e0-96ab-70c7da119904" (UID: "57c8ebc5-ed58-43e0-96ab-70c7da119904"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.911861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.911387    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-socket" pod "57c8ebc5-ed58-43e0-96ab-70c7da119904" (UID: "57c8ebc5-ed58-43e0-96ab-70c7da119904"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.911861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.911409    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-credential-socket" pod "57c8ebc5-ed58-43e0-96ab-70c7da119904" (UID: "57c8ebc5-ed58-43e0-96ab-70c7da119904"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.911861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.911611    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-data" pod "57c8ebc5-ed58-43e0-96ab-70c7da119904" (UID: "57c8ebc5-ed58-43e0-96ab-70c7da119904"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.911861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.911757    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.911861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.911778    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.911861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.911794    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.911861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.911810    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c8ebc5-ed58-43e0-96ab-70c7da119904-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.911861 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.911824    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:23.915672 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.915541    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-token" pod "57c8ebc5-ed58-43e0-96ab-70c7da119904" (UID: "57c8ebc5-ed58-43e0-96ab-70c7da119904"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:23.916150 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.916128    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-kube-api-access-jpzzw" pod "57c8ebc5-ed58-43e0-96ab-70c7da119904" (UID: "57c8ebc5-ed58-43e0-96ab-70c7da119904"). InnerVolumeSpecName "kube-api-access-jpzzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:23.916694 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.916659    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-envoy" pod "57c8ebc5-ed58-43e0-96ab-70c7da119904" (UID: "57c8ebc5-ed58-43e0-96ab-70c7da119904"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:23.916779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.916735    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-podinfo" pod "57c8ebc5-ed58-43e0-96ab-70c7da119904" (UID: "57c8ebc5-ed58-43e0-96ab-70c7da119904"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:23.925213 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.925192    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:34:23.959797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:23.959773    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:34:24.012300 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012268    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a9701a-c232-4626-938e-606c4ebfc5db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a9701a-c232-4626-938e-606c4ebfc5db-istiod-ca-cert\") pod \"67a9701a-c232-4626-938e-606c4ebfc5db\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") "
Nov 03 13:34:24.012300 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012309    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-token\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-token\") pod \"195f6efa-9355-46af-9a6a-9f857a1dedb6\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") "
Nov 03 13:34:24.012625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012335    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-socket\") pod \"195f6efa-9355-46af-9a6a-9f857a1dedb6\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") "
Nov 03 13:34:24.012625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012391    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-kube-api-access-lz668\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-kube-api-access-lz668\") pod \"195f6efa-9355-46af-9a6a-9f857a1dedb6\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") "
Nov 03 13:34:24.012625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012422    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-credential-socket\") pod \"67a9701a-c232-4626-938e-606c4ebfc5db\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") "
Nov 03 13:34:24.012625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012453    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-credential-socket\") pod \"195f6efa-9355-46af-9a6a-9f857a1dedb6\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") "
Nov 03 13:34:24.012625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012477    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-envoy\") pod \"67a9701a-c232-4626-938e-606c4ebfc5db\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") "
Nov 03 13:34:24.012625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012539    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-data\") pod \"195f6efa-9355-46af-9a6a-9f857a1dedb6\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") "
Nov 03 13:34:24.012625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012580    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-socket\") pod \"67a9701a-c232-4626-938e-606c4ebfc5db\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") "
Nov 03 13:34:24.012963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012640    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195f6efa-9355-46af-9a6a-9f857a1dedb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f6efa-9355-46af-9a6a-9f857a1dedb6-istiod-ca-cert\") pod \"195f6efa-9355-46af-9a6a-9f857a1dedb6\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") "
Nov 03 13:34:24.012963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012672    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67a9701a-c232-4626-938e-606c4ebfc5db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a9701a-c232-4626-938e-606c4ebfc5db-istio-podinfo\") pod \"67a9701a-c232-4626-938e-606c4ebfc5db\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") "
Nov 03 13:34:24.012963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012701    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-envoy\") pod \"195f6efa-9355-46af-9a6a-9f857a1dedb6\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") "
Nov 03 13:34:24.012963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012736    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-certs\") pod \"67a9701a-c232-4626-938e-606c4ebfc5db\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") "
Nov 03 13:34:24.012963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012799    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-certs\") pod \"195f6efa-9355-46af-9a6a-9f857a1dedb6\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") "
Nov 03 13:34:24.012963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012820    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-data\") pod \"67a9701a-c232-4626-938e-606c4ebfc5db\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") "
Nov 03 13:34:24.012963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012846    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-kube-api-access-9xfn2\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-kube-api-access-9xfn2\") pod \"67a9701a-c232-4626-938e-606c4ebfc5db\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") "
Nov 03 13:34:24.012963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012952    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-istio-token\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-istio-token\") pod \"67a9701a-c232-4626-938e-606c4ebfc5db\" (UID: \"67a9701a-c232-4626-938e-606c4ebfc5db\") "
Nov 03 13:34:24.013332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.012983    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-podinfo\") pod \"195f6efa-9355-46af-9a6a-9f857a1dedb6\" (UID: \"195f6efa-9355-46af-9a6a-9f857a1dedb6\") "
Nov 03 13:34:24.013332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.013275    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195f6efa-9355-46af-9a6a-9f857a1dedb6-istiod-ca-cert" pod "195f6efa-9355-46af-9a6a-9f857a1dedb6" (UID: "195f6efa-9355-46af-9a6a-9f857a1dedb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:24.013755 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.013732    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f6efa-9355-46af-9a6a-9f857a1dedb6-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.013836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.013759    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.013836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.013775    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.013836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.013789    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.013836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.013803    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpzzw\" (UniqueName: \"kubernetes.io/projected/57c8ebc5-ed58-43e0-96ab-70c7da119904-kube-api-access-jpzzw\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.014386 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.014075    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-credential-socket" pod "67a9701a-c232-4626-938e-606c4ebfc5db" (UID: "67a9701a-c232-4626-938e-606c4ebfc5db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.014386 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.014253    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-credential-socket" pod "195f6efa-9355-46af-9a6a-9f857a1dedb6" (UID: "195f6efa-9355-46af-9a6a-9f857a1dedb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.015901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.014941    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-certs" pod "195f6efa-9355-46af-9a6a-9f857a1dedb6" (UID: "195f6efa-9355-46af-9a6a-9f857a1dedb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.015901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.015249    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-socket" pod "195f6efa-9355-46af-9a6a-9f857a1dedb6" (UID: "195f6efa-9355-46af-9a6a-9f857a1dedb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.015901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.015445    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-data" pod "195f6efa-9355-46af-9a6a-9f857a1dedb6" (UID: "195f6efa-9355-46af-9a6a-9f857a1dedb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.016624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.016598    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a9701a-c232-4626-938e-606c4ebfc5db-istiod-ca-cert" pod "67a9701a-c232-4626-938e-606c4ebfc5db" (UID: "67a9701a-c232-4626-938e-606c4ebfc5db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:24.017835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.017806    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-data" pod "67a9701a-c232-4626-938e-606c4ebfc5db" (UID: "67a9701a-c232-4626-938e-606c4ebfc5db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.021457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.020778    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-podinfo" pod "195f6efa-9355-46af-9a6a-9f857a1dedb6" (UID: "195f6efa-9355-46af-9a6a-9f857a1dedb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:24.021457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.020826    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-envoy" pod "67a9701a-c232-4626-938e-606c4ebfc5db" (UID: "67a9701a-c232-4626-938e-606c4ebfc5db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.021457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.021032    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-socket" pod "67a9701a-c232-4626-938e-606c4ebfc5db" (UID: "67a9701a-c232-4626-938e-606c4ebfc5db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.021457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.021208    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-certs" pod "67a9701a-c232-4626-938e-606c4ebfc5db" (UID: "67a9701a-c232-4626-938e-606c4ebfc5db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.021457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.021217    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-token" pod "195f6efa-9355-46af-9a6a-9f857a1dedb6" (UID: "195f6efa-9355-46af-9a6a-9f857a1dedb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:24.021457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.021387    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-kube-api-access-lz668" pod "195f6efa-9355-46af-9a6a-9f857a1dedb6" (UID: "195f6efa-9355-46af-9a6a-9f857a1dedb6"). InnerVolumeSpecName "kube-api-access-lz668". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:24.021804 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.021752    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-envoy" pod "195f6efa-9355-46af-9a6a-9f857a1dedb6" (UID: "195f6efa-9355-46af-9a6a-9f857a1dedb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:24.022015 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.021992    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-istio-token" pod "67a9701a-c232-4626-938e-606c4ebfc5db" (UID: "67a9701a-c232-4626-938e-606c4ebfc5db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:24.022387 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.022330    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-kube-api-access-9xfn2" pod "67a9701a-c232-4626-938e-606c4ebfc5db" (UID: "67a9701a-c232-4626-938e-606c4ebfc5db"). InnerVolumeSpecName "kube-api-access-9xfn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:24.022580 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.022566    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67a9701a-c232-4626-938e-606c4ebfc5db-istio-podinfo" pod "67a9701a-c232-4626-938e-606c4ebfc5db" (UID: "67a9701a-c232-4626-938e-606c4ebfc5db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115364    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115404    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115419    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a9701a-c232-4626-938e-606c4ebfc5db-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115430    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115442    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115453    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115465    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115477    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xfn2\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-kube-api-access-9xfn2\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115494    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a9701a-c232-4626-938e-606c4ebfc5db-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115509    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115524    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a9701a-c232-4626-938e-606c4ebfc5db-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115541    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115552    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115573    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz668\" (UniqueName: \"kubernetes.io/projected/195f6efa-9355-46af-9a6a-9f857a1dedb6-kube-api-access-lz668\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115643    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115679    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f6efa-9355-46af-9a6a-9f857a1dedb6-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.115824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.115689    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a9701a-c232-4626-938e-606c4ebfc5db-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:24.195777 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.195735    4447 generic.go:356] "Generic (PLEG): container finished" podID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerID="71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf" exitCode=0
Nov 03 13:34:24.195968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.195901    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" event={"ID":"195f6efa-9355-46af-9a6a-9f857a1dedb6","Type":"ContainerDied","Data":"71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf"}
Nov 03 13:34:24.195968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.195951    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f" event={"ID":"195f6efa-9355-46af-9a6a-9f857a1dedb6","Type":"ContainerDied","Data":"e57910383057b6beb27a57d9b6f117122f43c97b9d93c5fd40d082e29241d217"}
Nov 03 13:34:24.196078 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.195979    4447 scope.go:122] "RemoveContainer" containerID="71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf"
Nov 03 13:34:24.196151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.196136    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"
Nov 03 13:34:24.199200 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.199168    4447 generic.go:356] "Generic (PLEG): container finished" podID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerID="e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a" exitCode=0
Nov 03 13:34:24.199380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.199270    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" event={"ID":"57c8ebc5-ed58-43e0-96ab-70c7da119904","Type":"ContainerDied","Data":"e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a"}
Nov 03 13:34:24.199380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.199313    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k" event={"ID":"57c8ebc5-ed58-43e0-96ab-70c7da119904","Type":"ContainerDied","Data":"1c7ecc99b99dc4e4fcbf5ad21462a4f0dee9fa17c90d9147f0b1e2d81ba04fdc"}
Nov 03 13:34:24.199552 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.199530    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"
Nov 03 13:34:24.205616 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.205589    4447 generic.go:356] "Generic (PLEG): container finished" podID="67a9701a-c232-4626-938e-606c4ebfc5db" containerID="16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f" exitCode=0
Nov 03 13:34:24.205813 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.205794    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" event={"ID":"67a9701a-c232-4626-938e-606c4ebfc5db","Type":"ContainerDied","Data":"16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f"}
Nov 03 13:34:24.205894 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.205838    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz" event={"ID":"67a9701a-c232-4626-938e-606c4ebfc5db","Type":"ContainerDied","Data":"e2822180ebb17fd63ef56d52d7416bb2bfd7696902ddad140b189eb03a6f6ad4"}
Nov 03 13:34:24.205952 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.205908    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"
Nov 03 13:34:24.208273 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.208229    4447 generic.go:356] "Generic (PLEG): container finished" podID="b6043f9d-679a-490b-8753-b32e618cce8c" containerID="cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68" exitCode=0
Nov 03 13:34:24.208333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.208271    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" event={"ID":"b6043f9d-679a-490b-8753-b32e618cce8c","Type":"ContainerDied","Data":"cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68"}
Nov 03 13:34:24.208333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.208303    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw" event={"ID":"b6043f9d-679a-490b-8753-b32e618cce8c","Type":"ContainerDied","Data":"1b1371ba102d2a47e8e15360772a263dc34b5cc7ce0f6545e6b269d05288e1c9"}
Nov 03 13:34:24.208466 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.208361    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"
Nov 03 13:34:24.228412 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.228386    4447 scope.go:122] "RemoveContainer" containerID="71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf"
Nov 03 13:34:24.229033 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:24.229011    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf\": container with ID starting with 71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf not found: ID does not exist" containerID="71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf"
Nov 03 13:34:24.229262 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.229047    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf"} err="failed to get container status \"71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf\": rpc error: code = NotFound desc = could not find container \"71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf\": container with ID starting with 71ebdb28192194762a23578edee12bd8a210d3623c412a85259a38779955cbdf not found: ID does not exist"
Nov 03 13:34:24.229447 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.229079    4447 scope.go:122] "RemoveContainer" containerID="e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a"
Nov 03 13:34:24.232757 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.232725    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"]
Nov 03 13:34:24.237012 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.236990    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fgd4f"]
Nov 03 13:34:24.240328 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.240312    4447 scope.go:122] "RemoveContainer" containerID="e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a"
Nov 03 13:34:24.240720 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:24.240703    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a\": container with ID starting with e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a not found: ID does not exist" containerID="e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a"
Nov 03 13:34:24.240759 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.240728    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a"} err="failed to get container status \"e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a\": rpc error: code = NotFound desc = could not find container \"e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a\": container with ID starting with e5ae42a2602c23c38cae836e1ff347764c1c677962b0727b0464d1a10037b30a not found: ID does not exist"
Nov 03 13:34:24.240759 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.240745    4447 scope.go:122] "RemoveContainer" containerID="16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f"
Nov 03 13:34:24.250604 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.250585    4447 scope.go:122] "RemoveContainer" containerID="16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f"
Nov 03 13:34:24.250950 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:24.250930    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f\": container with ID starting with 16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f not found: ID does not exist" containerID="16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f"
Nov 03 13:34:24.251015 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.250958    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f"} err="failed to get container status \"16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f\": rpc error: code = NotFound desc = could not find container \"16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f\": container with ID starting with 16c051b3912b2cba3aded86fb10e823fd7c3ed8d554f71529bbfd867e690136f not found: ID does not exist"
Nov 03 13:34:24.251015 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.250977    4447 scope.go:122] "RemoveContainer" containerID="cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68"
Nov 03 13:34:24.254435 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.254414    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"]
Nov 03 13:34:24.262504 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.262484    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"]
Nov 03 13:34:24.263065 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263052    4447 cpu_manager.go:414] "containerMap: removing container" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263065    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263080    4447 cpu_manager.go:414] "containerMap: removing container" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263085    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263093    4447 cpu_manager.go:414] "containerMap: removing container" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263098    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263105    4447 cpu_manager.go:414] "containerMap: removing container" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263110    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263119    4447 cpu_manager.go:414] "containerMap: removing container" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263123    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263130    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263134    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263142    4447 cpu_manager.go:414] "containerMap: removing container" podUID="aa0ccbe3-8de9-4c60-9448-67dab863b805" containerName="maas-api"
Nov 03 13:34:24.263148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263147    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0ccbe3-8de9-4c60-9448-67dab863b805" containerName="maas-api"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263194    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263201    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263289    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="484ecf48-e4c0-4175-b9b3-7a03e31bc001" containerName="istio-proxy"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263299    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" containerName="istio-proxy"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263306    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="db6c06c3-3ebc-413f-b9f2-3295277b123c" containerName="istio-proxy"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263312    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" containerName="istio-proxy"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263319    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" containerName="istio-proxy"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263324    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" containerName="istio-proxy"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263330    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0ccbe3-8de9-4c60-9448-67dab863b805" containerName="maas-api"
Nov 03 13:34:24.263778 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.263358    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4036ac2c-02b8-4eed-9efa-59b749a95b8b" containerName="istio-proxy"
Nov 03 13:34:24.268323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.268307    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-qrf2k"]
Nov 03 13:34:24.268406 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.268397    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:24.276676 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.275807    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a724a6\""
Nov 03 13:34:24.276808 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.276735    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"]
Nov 03 13:34:24.280321 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.280303    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"]
Nov 03 13:34:24.287689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.287665    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lksnz"]
Nov 03 13:34:24.290589 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.290523    4447 scope.go:122] "RemoveContainer" containerID="cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68"
Nov 03 13:34:24.291091 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:24.291058    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68\": container with ID starting with cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68 not found: ID does not exist" containerID="cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68"
Nov 03 13:34:24.291210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.291099    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68"} err="failed to get container status \"cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68\": rpc error: code = NotFound desc = could not find container \"cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68\": container with ID starting with cc706dab3d573c24121c2d9a08e4ad4ab133691703df15b5debd72379117ff68 not found: ID does not exist"
Nov 03 13:34:24.304014 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.303971    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"]
Nov 03 13:34:24.309126 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.309054    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"]
Nov 03 13:34:24.310069 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.310049    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:24.325486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.323621    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a724a6-dockercfg-qw5z6\""
Nov 03 13:34:24.335964 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.335932    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llcj\" (UniqueName: \"kubernetes.io/projected/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-kube-api-access-7llcj\") pod \"payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9\" (UID: \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:24.340402 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.340370    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9\" (UID: \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:24.342638 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.342617    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195f6efa-9355-46af-9a6a-9f857a1dedb6" path="/var/lib/kubelet/pods/195f6efa-9355-46af-9a6a-9f857a1dedb6/volumes"
Nov 03 13:34:24.343303 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.343282    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c8ebc5-ed58-43e0-96ab-70c7da119904" path="/var/lib/kubelet/pods/57c8ebc5-ed58-43e0-96ab-70c7da119904/volumes"
Nov 03 13:34:24.343904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.343885    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a9701a-c232-4626-938e-606c4ebfc5db" path="/var/lib/kubelet/pods/67a9701a-c232-4626-938e-606c4ebfc5db/volumes"
Nov 03 13:34:24.344832 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.344804    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"]
Nov 03 13:34:24.344928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.344879    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-jv6zw"]
Nov 03 13:34:24.450370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.448169    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llcj\" (UniqueName: \"kubernetes.io/projected/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-kube-api-access-7llcj\") pod \"payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9\" (UID: \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:24.450370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.449516    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9\" (UID: \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:24.450370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.449743    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9\" (UID: \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:24.450370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.449987    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6efbf-968c-4e5d-93b3-a62eaae802d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz\" (UID: \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:24.450370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.450075    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bd8g\" (UniqueName: \"kubernetes.io/projected/21a6efbf-968c-4e5d-93b3-a62eaae802d6-kube-api-access-9bd8g\") pod \"payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz\" (UID: \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:24.465852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.461752    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llcj\" (UniqueName: \"kubernetes.io/projected/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-kube-api-access-7llcj\") pod \"payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9\" (UID: \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:24.551702 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.551665    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6efbf-968c-4e5d-93b3-a62eaae802d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz\" (UID: \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:24.552091 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.551761    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bd8g\" (UniqueName: \"kubernetes.io/projected/21a6efbf-968c-4e5d-93b3-a62eaae802d6-kube-api-access-9bd8g\") pod \"payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz\" (UID: \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:24.552572 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.552546    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6efbf-968c-4e5d-93b3-a62eaae802d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz\" (UID: \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:24.562259 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.562230    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bd8g\" (UniqueName: \"kubernetes.io/projected/21a6efbf-968c-4e5d-93b3-a62eaae802d6-kube-api-access-9bd8g\") pod \"payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz\" (UID: \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:24.606244 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.606212    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:24.643756 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.643484    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:24.905297 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.905269    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"]
Nov 03 13:34:24.906691 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:34:24.906664    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cb9a4b_78ab_4a95_a88b_a9a2f83fac28.slice/crio-524e2c386a780e677c275eab928f0d2e52d0825f800e1bc6fac493c17c1f0c3b WatchSource:0}: Error finding container 524e2c386a780e677c275eab928f0d2e52d0825f800e1bc6fac493c17c1f0c3b: Status 404 returned error can't find the container with id 524e2c386a780e677c275eab928f0d2e52d0825f800e1bc6fac493c17c1f0c3b
Nov 03 13:34:24.965370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:24.965321    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"]
Nov 03 13:34:24.970584 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:34:24.970558    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a6efbf_968c_4e5d_93b3_a62eaae802d6.slice/crio-e3a02de32fe96504b813fc22f9ea87b8e4adc8b12990a40b92bf90a3125d956e WatchSource:0}: Error finding container e3a02de32fe96504b813fc22f9ea87b8e4adc8b12990a40b92bf90a3125d956e: Status 404 returned error can't find the container with id e3a02de32fe96504b813fc22f9ea87b8e4adc8b12990a40b92bf90a3125d956e
Nov 03 13:34:25.218931 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:25.218895    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9" event={"ID":"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28","Type":"ContainerStarted","Data":"29c9a42920479721a116b273eac9bf5e995662db831f2532e6026922fe1efc5d"}
Nov 03 13:34:25.219130 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:25.218949    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9" event={"ID":"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28","Type":"ContainerStarted","Data":"524e2c386a780e677c275eab928f0d2e52d0825f800e1bc6fac493c17c1f0c3b"}
Nov 03 13:34:25.219130 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:25.219077    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:25.220667 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:25.220640    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz" event={"ID":"21a6efbf-968c-4e5d-93b3-a62eaae802d6","Type":"ContainerStarted","Data":"6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd"}
Nov 03 13:34:25.220797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:25.220685    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz" event={"ID":"21a6efbf-968c-4e5d-93b3-a62eaae802d6","Type":"ContainerStarted","Data":"e3a02de32fe96504b813fc22f9ea87b8e4adc8b12990a40b92bf90a3125d956e"}
Nov 03 13:34:25.220797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:25.220764    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:25.242112 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:25.242009    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9" podStartSLOduration=1.2419936950000001 podStartE2EDuration="1.241993695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:25.239241567 +0000 UTC m=+1553.050909783" watchObservedRunningTime="2026-11-03 13:34:25.241993695 +0000 UTC m=+1553.053661882"
Nov 03 13:34:25.278491 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:25.278427    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz" podStartSLOduration=1.278413387 podStartE2EDuration="1.278413387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:25.275577856 +0000 UTC m=+1553.087246047" watchObservedRunningTime="2026-11-03 13:34:25.278413387 +0000 UTC m=+1553.090081574"
Nov 03 13:34:26.320408 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:26.320376    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6043f9d-679a-490b-8753-b32e618cce8c" path="/var/lib/kubelet/pods/b6043f9d-679a-490b-8753-b32e618cce8c/volumes"
Nov 03 13:34:26.746087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:26.746049    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-d752f5"]
Nov 03 13:34:26.751568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:26.751165    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:26.754307 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:26.754284    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncr7v\""
Nov 03 13:34:26.772476 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:26.772442    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-d752f5"]
Nov 03 13:34:26.886913 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:26.886874    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8hd\" (UniqueName: \"kubernetes.io/projected/cedbcde9-6111-44c1-aadf-063e0f716ff0-kube-api-access-xr8hd\") pod \"test-curl-7092-d752f5\" (UID: \"cedbcde9-6111-44c1-aadf-063e0f716ff0\") " pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:26.988635 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:26.988599    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8hd\" (UniqueName: \"kubernetes.io/projected/cedbcde9-6111-44c1-aadf-063e0f716ff0-kube-api-access-xr8hd\") pod \"test-curl-7092-d752f5\" (UID: \"cedbcde9-6111-44c1-aadf-063e0f716ff0\") " pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:26.996983 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:26.996923    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8hd\" (UniqueName: \"kubernetes.io/projected/cedbcde9-6111-44c1-aadf-063e0f716ff0-kube-api-access-xr8hd\") pod \"test-curl-7092-d752f5\" (UID: \"cedbcde9-6111-44c1-aadf-063e0f716ff0\") " pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:27.075640 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:27.075607    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:27.354540 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:27.354508    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-d752f5"]
Nov 03 13:34:27.355600 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:34:27.355572    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcedbcde9_6111_44c1_aadf_063e0f716ff0.slice/crio-6aedb07c629cc4b7476b2b564cd52bbdaca158b4b867a71844339e48faae696d WatchSource:0}: Error finding container 6aedb07c629cc4b7476b2b564cd52bbdaca158b4b867a71844339e48faae696d: Status 404 returned error can't find the container with id 6aedb07c629cc4b7476b2b564cd52bbdaca158b4b867a71844339e48faae696d
Nov 03 13:34:28.245597 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:28.245552    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-d752f5" event={"ID":"cedbcde9-6111-44c1-aadf-063e0f716ff0","Type":"ContainerStarted","Data":"4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36"}
Nov 03 13:34:28.245775 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:28.245606    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-d752f5" event={"ID":"cedbcde9-6111-44c1-aadf-063e0f716ff0","Type":"ContainerStarted","Data":"6aedb07c629cc4b7476b2b564cd52bbdaca158b4b867a71844339e48faae696d"}
Nov 03 13:34:28.262437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:28.262389    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7092-d752f5" podStartSLOduration=2.26237546 podStartE2EDuration="2.26237546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:28.259233291 +0000 UTC m=+1556.070901515" watchObservedRunningTime="2026-11-03 13:34:28.26237546 +0000 UTC m=+1556.074043648"
Nov 03 13:34:28.954685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:28.954642    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-d752f5"]
Nov 03 13:34:28.962395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:28.962363    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-d752f5"]
Nov 03 13:34:30.260538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:30.260496    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7092-d752f5" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" containerName="test-curl-7092-d752f5" containerID="cri-o://4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36" gracePeriod=2
Nov 03 13:34:30.262296 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:30.262263    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-d752f5\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:32.499966 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:32.499937    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:32.501794 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:32.501762    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-d752f5\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:32.564624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:32.564549    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cedbcde9-6111-44c1-aadf-063e0f716ff0-kube-api-access-xr8hd\" (UniqueName: \"kubernetes.io/projected/cedbcde9-6111-44c1-aadf-063e0f716ff0-kube-api-access-xr8hd\") pod \"cedbcde9-6111-44c1-aadf-063e0f716ff0\" (UID: \"cedbcde9-6111-44c1-aadf-063e0f716ff0\") "
Nov 03 13:34:32.566736 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:32.566708    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedbcde9-6111-44c1-aadf-063e0f716ff0-kube-api-access-xr8hd" pod "cedbcde9-6111-44c1-aadf-063e0f716ff0" (UID: "cedbcde9-6111-44c1-aadf-063e0f716ff0"). InnerVolumeSpecName "kube-api-access-xr8hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:32.666776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:32.666735    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8hd\" (UniqueName: \"kubernetes.io/projected/cedbcde9-6111-44c1-aadf-063e0f716ff0-kube-api-access-xr8hd\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:33.281480 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.281434    4447 generic.go:356] "Generic (PLEG): container finished" podID="cedbcde9-6111-44c1-aadf-063e0f716ff0" containerID="4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36" exitCode=137
Nov 03 13:34:33.281635 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.281505    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:33.281635 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.281534    4447 scope.go:122] "RemoveContainer" containerID="4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36"
Nov 03 13:34:33.283571 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:33.283544    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-d752f5\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:33.292118 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.292096    4447 scope.go:122] "RemoveContainer" containerID="4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36"
Nov 03 13:34:33.292398 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:33.292379    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36\": container with ID starting with 4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36 not found: ID does not exist" containerID="4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36"
Nov 03 13:34:33.292452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.292408    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36"} err="failed to get container status \"4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36\": rpc error: code = NotFound desc = could not find container \"4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36\": container with ID starting with 4f2f40180e0a2184d48f942386aedb213e11f8489c9a58a468cd637e89af5f36 not found: ID does not exist"
Nov 03 13:34:33.292678 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:33.292659    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-d752f5\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:33.411900 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:33.411865    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:34:33.633487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.633407    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"]
Nov 03 13:34:33.634404 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.634382    4447 cpu_manager.go:414] "containerMap: removing container" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" containerName="test-curl-7092-d752f5"
Nov 03 13:34:33.634497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.634405    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" containerName="test-curl-7092-d752f5"
Nov 03 13:34:33.634655 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.634640    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" containerName="test-curl-7092-d752f5"
Nov 03 13:34:33.640783 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.640759    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.648396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.648082    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-a724a6-serving-cert\""
Nov 03 13:34:33.650302 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.650273    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"]
Nov 03 13:34:33.667593 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:33.667219    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-d752f5\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" pod="opendatahub/test-curl-7092-d752f5"
Nov 03 13:34:33.679993 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.679966    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.680131 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.680039    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.680256 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.680226    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.680370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.680336    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nklk\" (UniqueName: \"kubernetes.io/projected/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-kube-api-access-8nklk\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.781627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.781587    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.781817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.781657    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nklk\" (UniqueName: \"kubernetes.io/projected/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-kube-api-access-8nklk\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.781817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.781781    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.781924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.781848    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.782307 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.782283    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.784278 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.784255    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.784377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.784280    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.790503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.790457    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nklk\" (UniqueName: \"kubernetes.io/projected/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-kube-api-access-8nklk\") pod \"maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:33.975017 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:33.974979    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:34.303419 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:34.300502    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"]
Nov 03 13:34:34.308810 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:34:34.308780    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f40f2b_a4bd_4aae_8e18_b88afd5e5c7e.slice/crio-6da4b30eaa328b4ecbfa13675ae757bab9b0cb13b218ec920f67242bec946e0b WatchSource:0}: Error finding container 6da4b30eaa328b4ecbfa13675ae757bab9b0cb13b218ec920f67242bec946e0b: Status 404 returned error can't find the container with id 6da4b30eaa328b4ecbfa13675ae757bab9b0cb13b218ec920f67242bec946e0b
Nov 03 13:34:34.322619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:34.322593    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cedbcde9-6111-44c1-aadf-063e0f716ff0" path="/var/lib/kubelet/pods/cedbcde9-6111-44c1-aadf-063e0f716ff0/volumes"
Nov 03 13:34:34.929593 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:34.929559    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"]
Nov 03 13:34:35.303783 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:35.303743    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65" event={"ID":"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e","Type":"ContainerStarted","Data":"7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54"}
Nov 03 13:34:35.303957 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:35.303804    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65" event={"ID":"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e","Type":"ContainerStarted","Data":"6da4b30eaa328b4ecbfa13675ae757bab9b0cb13b218ec920f67242bec946e0b"}
Nov 03 13:34:35.303957 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:35.303834    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:35.321787 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:35.321743    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65" podStartSLOduration=1.833507668 podStartE2EDuration="2.321730725s" totalImagesPullingTime="488.223057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:35.321181524 +0000 UTC m=+1563.132849733" watchObservedRunningTime="2026-11-03 13:34:35.321730725 +0000 UTC m=+1563.133398913"
Nov 03 13:34:36.232968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:36.232942    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:34:36.233437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:36.233014    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:34:39.930283 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:39.930238    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4" podUID="d6c794c4-b377-488d-8042-4efb90f80071" containerName="payload-processing" containerID="cri-o://bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5" gracePeriod=25
Nov 03 13:34:40.235992 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.235655    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:34:40.293589 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.281009    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-provider-credentials\") pod \"d6c794c4-b377-488d-8042-4efb90f80071\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") "
Nov 03 13:34:40.297669 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.297633    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-kube-api-access-nf9jv\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-kube-api-access-nf9jv\") pod \"d6c794c4-b377-488d-8042-4efb90f80071\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") "
Nov 03 13:34:40.302880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.302851    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-routing-overlay-volume\") pod \"d6c794c4-b377-488d-8042-4efb90f80071\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") "
Nov 03 13:34:40.303153 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.303131    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-plugins-config-volume\") pod \"d6c794c4-b377-488d-8042-4efb90f80071\" (UID: \"d6c794c4-b377-488d-8042-4efb90f80071\") "
Nov 03 13:34:40.304863 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.304836    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-plugins-config-volume" pod "d6c794c4-b377-488d-8042-4efb90f80071" (UID: "d6c794c4-b377-488d-8042-4efb90f80071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:40.305373 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.305297    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-routing-overlay-volume" pod "d6c794c4-b377-488d-8042-4efb90f80071" (UID: "d6c794c4-b377-488d-8042-4efb90f80071"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:40.332214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.331538    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-provider-credentials" pod "d6c794c4-b377-488d-8042-4efb90f80071" (UID: "d6c794c4-b377-488d-8042-4efb90f80071"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:40.336929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.336878    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-kube-api-access-nf9jv" pod "d6c794c4-b377-488d-8042-4efb90f80071" (UID: "d6c794c4-b377-488d-8042-4efb90f80071"). InnerVolumeSpecName "kube-api-access-nf9jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:40.355041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.347479    4447 generic.go:356] "Generic (PLEG): container finished" podID="d6c794c4-b377-488d-8042-4efb90f80071" containerID="bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5" exitCode=0
Nov 03 13:34:40.355041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.347539    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4" event={"ID":"d6c794c4-b377-488d-8042-4efb90f80071","Type":"ContainerDied","Data":"bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5"}
Nov 03 13:34:40.355041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.347606    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4" event={"ID":"d6c794c4-b377-488d-8042-4efb90f80071","Type":"ContainerDied","Data":"576b48024f941f465ab5d9c35668d4fcacc4ef8d6225502320259582dd1f445d"}
Nov 03 13:34:40.355041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.347636    4447 scope.go:122] "RemoveContainer" containerID="bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5"
Nov 03 13:34:40.355041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.349848    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"
Nov 03 13:34:40.365822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.365797    4447 scope.go:122] "RemoveContainer" containerID="bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5"
Nov 03 13:34:40.366213 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:34:40.366190    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5\": container with ID starting with bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5 not found: ID does not exist" containerID="bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5"
Nov 03 13:34:40.366300 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.366219    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5"} err="failed to get container status \"bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5\": rpc error: code = NotFound desc = could not find container \"bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5\": container with ID starting with bce50264ea43ca4fac146d3b58fae84ad445908d8a4ce5fd14f216fa218363d5 not found: ID does not exist"
Nov 03 13:34:40.399276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.399224    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"]
Nov 03 13:34:40.404968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.404933    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9jv\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-kube-api-access-nf9jv\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:40.405151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.404981    4447 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-routing-overlay-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:40.405151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.404995    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c794c4-b377-488d-8042-4efb90f80071-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:40.405151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.405008    4447 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6c794c4-b377-488d-8042-4efb90f80071-provider-credentials\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:40.405667 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:40.405641    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd5nvv4"]
Nov 03 13:34:41.319592 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:41.319565    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:34:42.325822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:42.325773    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c794c4-b377-488d-8042-4efb90f80071" path="/var/lib/kubelet/pods/d6c794c4-b377-488d-8042-4efb90f80071/volumes"
Nov 03 13:34:53.838956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.838870    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"]
Nov 03 13:34:53.839748 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.839727    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d6c794c4-b377-488d-8042-4efb90f80071" containerName="payload-processing"
Nov 03 13:34:53.839866 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.839748    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c794c4-b377-488d-8042-4efb90f80071" containerName="payload-processing"
Nov 03 13:34:53.839929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.839885    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c794c4-b377-488d-8042-4efb90f80071" containerName="payload-processing"
Nov 03 13:34:53.843688 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.843667    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:34:53.849210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.849183    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-2b8a08-dockercfg-8m7lr\""
Nov 03 13:34:53.849325 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.849219    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-2b8a08\""
Nov 03 13:34:53.851363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.851279    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"]
Nov 03 13:34:53.856496 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.856474    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:34:53.857572 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.857540    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"]
Nov 03 13:34:53.875246 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.875204    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"]
Nov 03 13:34:53.945929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.945881    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7405dbd-4d03-4519-b204-bdb453592985-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz\" (UID: \"e7405dbd-4d03-4519-b204-bdb453592985\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:34:53.946597 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.946562    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9hv\" (UniqueName: \"kubernetes.io/projected/ca293290-1f26-4abf-a5ad-feccce1c7562-kube-api-access-qf9hv\") pod \"payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn\" (UID: \"ca293290-1f26-4abf-a5ad-feccce1c7562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:34:53.946721 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.946687    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca293290-1f26-4abf-a5ad-feccce1c7562-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn\" (UID: \"ca293290-1f26-4abf-a5ad-feccce1c7562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:34:53.946786 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:53.946757    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq46f\" (UniqueName: \"kubernetes.io/projected/e7405dbd-4d03-4519-b204-bdb453592985-kube-api-access-zq46f\") pod \"payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz\" (UID: \"e7405dbd-4d03-4519-b204-bdb453592985\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:34:54.048591 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.048553    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7405dbd-4d03-4519-b204-bdb453592985-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz\" (UID: \"e7405dbd-4d03-4519-b204-bdb453592985\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:34:54.048809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.048774    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9hv\" (UniqueName: \"kubernetes.io/projected/ca293290-1f26-4abf-a5ad-feccce1c7562-kube-api-access-qf9hv\") pod \"payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn\" (UID: \"ca293290-1f26-4abf-a5ad-feccce1c7562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:34:54.048881 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.048824    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca293290-1f26-4abf-a5ad-feccce1c7562-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn\" (UID: \"ca293290-1f26-4abf-a5ad-feccce1c7562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:34:54.048881 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.048874    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq46f\" (UniqueName: \"kubernetes.io/projected/e7405dbd-4d03-4519-b204-bdb453592985-kube-api-access-zq46f\") pod \"payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz\" (UID: \"e7405dbd-4d03-4519-b204-bdb453592985\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:34:54.049459 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.049438    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7405dbd-4d03-4519-b204-bdb453592985-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz\" (UID: \"e7405dbd-4d03-4519-b204-bdb453592985\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:34:54.049674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.049652    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca293290-1f26-4abf-a5ad-feccce1c7562-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn\" (UID: \"ca293290-1f26-4abf-a5ad-feccce1c7562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:34:54.056557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.056532    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9hv\" (UniqueName: \"kubernetes.io/projected/ca293290-1f26-4abf-a5ad-feccce1c7562-kube-api-access-qf9hv\") pod \"payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn\" (UID: \"ca293290-1f26-4abf-a5ad-feccce1c7562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:34:54.056763 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.056740    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq46f\" (UniqueName: \"kubernetes.io/projected/e7405dbd-4d03-4519-b204-bdb453592985-kube-api-access-zq46f\") pod \"payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz\" (UID: \"e7405dbd-4d03-4519-b204-bdb453592985\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:34:54.163762 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.163645    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:34:54.181359 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.181316    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:34:54.457714 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.457689    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"]
Nov 03 13:34:54.460257 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:34:54.460226    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7405dbd_4d03_4519_b204_bdb453592985.slice/crio-681c096dd50946585cc9195eb43a87cb586cfdfcedec2eb612128c3683067c31 WatchSource:0}: Error finding container 681c096dd50946585cc9195eb43a87cb586cfdfcedec2eb612128c3683067c31: Status 404 returned error can't find the container with id 681c096dd50946585cc9195eb43a87cb586cfdfcedec2eb612128c3683067c31
Nov 03 13:34:54.489000 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.488951    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"]
Nov 03 13:34:54.505309 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:54.505272    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz" event={"ID":"e7405dbd-4d03-4519-b204-bdb453592985","Type":"ContainerStarted","Data":"681c096dd50946585cc9195eb43a87cb586cfdfcedec2eb612128c3683067c31"}
Nov 03 13:34:55.514015 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:55.513975    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz" event={"ID":"e7405dbd-4d03-4519-b204-bdb453592985","Type":"ContainerStarted","Data":"1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9"}
Nov 03 13:34:55.514522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:55.514055    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:34:55.515770 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:55.515739    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn" event={"ID":"ca293290-1f26-4abf-a5ad-feccce1c7562","Type":"ContainerStarted","Data":"0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01"}
Nov 03 13:34:55.515905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:55.515788    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn" event={"ID":"ca293290-1f26-4abf-a5ad-feccce1c7562","Type":"ContainerStarted","Data":"70f746e800156ded84a7fb674c8559bad48d877ea326ef157f0eaa2f1a4983dc"}
Nov 03 13:34:55.515905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:55.515809    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:34:55.554035 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:34:55.553972    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz" podStartSLOduration=2.553956232 podStartE2EDuration="2.553956232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:55.532946926 +0000 UTC m=+1583.344615115" watchObservedRunningTime="2026-11-03 13:34:55.553956232 +0000 UTC m=+1583.365624420"
Nov 03 13:35:03.418755 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:03.418714    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:35:03.914717 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:03.914660    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn" podStartSLOduration=10.914641228 podStartE2EDuration="10.914641228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:55.551389269 +0000 UTC m=+1583.363057456" watchObservedRunningTime="2026-11-03 13:35:03.914641228 +0000 UTC m=+1591.726309418"
Nov 03 13:35:03.915184 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:03.915153    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"]
Nov 03 13:35:03.926380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:03.923155    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:03.932225 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:03.931105    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-2b8a08-serving-cert\""
Nov 03 13:35:03.935721 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:03.935695    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"]
Nov 03 13:35:04.012366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.002074    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d6076-2ccf-44c4-9eef-18efb41051b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.012366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.002157    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.012366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.002191    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.012366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.002232    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gsjc\" (UniqueName: \"kubernetes.io/projected/ea5d6076-2ccf-44c4-9eef-18efb41051b8-kube-api-access-5gsjc\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.103188 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.103145    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.103380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.103195    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.103380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.103236    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gsjc\" (UniqueName: \"kubernetes.io/projected/ea5d6076-2ccf-44c4-9eef-18efb41051b8-kube-api-access-5gsjc\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.103538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.103520    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d6076-2ccf-44c4-9eef-18efb41051b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.104290 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.104263    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d6076-2ccf-44c4-9eef-18efb41051b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.106025 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.106001    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.106181 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.106158    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.111968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.111937    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gsjc\" (UniqueName: \"kubernetes.io/projected/ea5d6076-2ccf-44c4-9eef-18efb41051b8-kube-api-access-5gsjc\") pod \"maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.255096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.255063    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:04.571213 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.571124    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"]
Nov 03 13:35:04.583118 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:35:04.583079    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5d6076_2ccf_44c4_9eef_18efb41051b8.slice/crio-67f21a106b3b9c0fd58903da6af8903201264bc82650c3428ca8d944e01e2c0a WatchSource:0}: Error finding container 67f21a106b3b9c0fd58903da6af8903201264bc82650c3428ca8d944e01e2c0a: Status 404 returned error can't find the container with id 67f21a106b3b9c0fd58903da6af8903201264bc82650c3428ca8d944e01e2c0a
Nov 03 13:35:04.904722 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.904634    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"]
Nov 03 13:35:04.904919 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:04.904892    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc" podUID="dd130566-cad8-41be-8096-66553b8420ac" containerName="maas-api" containerID="cri-o://137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d" gracePeriod=30
Nov 03 13:35:05.388524 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.388495    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:35:05.555919 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.555838    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd130566-cad8-41be-8096-66553b8420ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd130566-cad8-41be-8096-66553b8420ac-openshift-service-ca\") pod \"dd130566-cad8-41be-8096-66553b8420ac\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") "
Nov 03 13:35:05.555919 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.555915    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd130566-cad8-41be-8096-66553b8420ac-kube-api-access-6rnnb\" (UniqueName: \"kubernetes.io/projected/dd130566-cad8-41be-8096-66553b8420ac-kube-api-access-6rnnb\") pod \"dd130566-cad8-41be-8096-66553b8420ac\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") "
Nov 03 13:35:05.556141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.556060    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-maas-api-tls\") pod \"dd130566-cad8-41be-8096-66553b8420ac\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") "
Nov 03 13:35:05.556141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.556107    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-metrics-certs\") pod \"dd130566-cad8-41be-8096-66553b8420ac\" (UID: \"dd130566-cad8-41be-8096-66553b8420ac\") "
Nov 03 13:35:05.556386 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.556357    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd130566-cad8-41be-8096-66553b8420ac-openshift-service-ca" pod "dd130566-cad8-41be-8096-66553b8420ac" (UID: "dd130566-cad8-41be-8096-66553b8420ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:05.556814 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.556787    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd130566-cad8-41be-8096-66553b8420ac-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:05.558684 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.558655    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-metrics-certs" pod "dd130566-cad8-41be-8096-66553b8420ac" (UID: "dd130566-cad8-41be-8096-66553b8420ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:05.559121 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.559104    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-maas-api-tls" pod "dd130566-cad8-41be-8096-66553b8420ac" (UID: "dd130566-cad8-41be-8096-66553b8420ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:05.559274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.559248    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd130566-cad8-41be-8096-66553b8420ac-kube-api-access-6rnnb" pod "dd130566-cad8-41be-8096-66553b8420ac" (UID: "dd130566-cad8-41be-8096-66553b8420ac"). InnerVolumeSpecName "kube-api-access-6rnnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:05.603997 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.603965    4447 generic.go:356] "Generic (PLEG): container finished" podID="dd130566-cad8-41be-8096-66553b8420ac" containerID="137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d" exitCode=0
Nov 03 13:35:05.604505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.604014    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"
Nov 03 13:35:05.604505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.604055    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc" event={"ID":"dd130566-cad8-41be-8096-66553b8420ac","Type":"ContainerDied","Data":"137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d"}
Nov 03 13:35:05.604505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.604155    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc" event={"ID":"dd130566-cad8-41be-8096-66553b8420ac","Type":"ContainerDied","Data":"fbe063e72dbb32fbf9aaf4113ad20ce5868164a195f3e3abc2edd0deb7cbc7bb"}
Nov 03 13:35:05.604505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.604188    4447 scope.go:122] "RemoveContainer" containerID="137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d"
Nov 03 13:35:05.609394 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.606211    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb" event={"ID":"ea5d6076-2ccf-44c4-9eef-18efb41051b8","Type":"ContainerStarted","Data":"72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d"}
Nov 03 13:35:05.609394 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.606258    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb" event={"ID":"ea5d6076-2ccf-44c4-9eef-18efb41051b8","Type":"ContainerStarted","Data":"67f21a106b3b9c0fd58903da6af8903201264bc82650c3428ca8d944e01e2c0a"}
Nov 03 13:35:05.609394 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.606379    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:05.617975 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.617952    4447 scope.go:122] "RemoveContainer" containerID="137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d"
Nov 03 13:35:05.618335 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:05.618303    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d\": container with ID starting with 137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d not found: ID does not exist" containerID="137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d"
Nov 03 13:35:05.618445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.618359    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d"} err="failed to get container status \"137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d\": rpc error: code = NotFound desc = could not find container \"137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d\": container with ID starting with 137dc94829042eebffa674c7c26d3dbfd81df8062318ea2533b51bcff065099d not found: ID does not exist"
Nov 03 13:35:05.626663 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.626617    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb" podStartSLOduration=2.135285092 podStartE2EDuration="2.626604101s" totalImagesPullingTime="491.319009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:05.624824725 +0000 UTC m=+1593.436492935" watchObservedRunningTime="2026-11-03 13:35:05.626604101 +0000 UTC m=+1593.438272596"
Nov 03 13:35:05.643685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.643657    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"]
Nov 03 13:35:05.647440 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.647412    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-9swrc"]
Nov 03 13:35:05.658583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.658557    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnnb\" (UniqueName: \"kubernetes.io/projected/dd130566-cad8-41be-8096-66553b8420ac-kube-api-access-6rnnb\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:05.658583 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.658583    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:05.658745 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:05.658597    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd130566-cad8-41be-8096-66553b8420ac-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:06.325282 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:06.325248    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd130566-cad8-41be-8096-66553b8420ac" path="/var/lib/kubelet/pods/dd130566-cad8-41be-8096-66553b8420ac/volumes"
Nov 03 13:35:06.526901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:06.526874    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:35:06.527046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:06.526952    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:35:11.623535 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:11.623503    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:35:18.692075 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.692041    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"]
Nov 03 13:35:18.692968 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.692944    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dd130566-cad8-41be-8096-66553b8420ac" containerName="maas-api"
Nov 03 13:35:18.693093 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.692970    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd130566-cad8-41be-8096-66553b8420ac" containerName="maas-api"
Nov 03 13:35:18.693184 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.693170    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd130566-cad8-41be-8096-66553b8420ac" containerName="maas-api"
Nov 03 13:35:18.700362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.700319    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.722679 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.722646    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1ec4b892-serving-cert\""
Nov 03 13:35:18.729015 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.728983    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"]
Nov 03 13:35:18.782318 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.782275    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-metrics-certs\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.782573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.782422    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-maas-api-tls\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.782573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.782547    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8cv\" (UniqueName: \"kubernetes.io/projected/86c79098-7c03-49b1-a6a8-cb33818ce7ee-kube-api-access-7k8cv\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.782573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.782568    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c79098-7c03-49b1-a6a8-cb33818ce7ee-openshift-service-ca\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.883776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.883736    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-maas-api-tls\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.883985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.883803    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8cv\" (UniqueName: \"kubernetes.io/projected/86c79098-7c03-49b1-a6a8-cb33818ce7ee-kube-api-access-7k8cv\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.883985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.883911    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c79098-7c03-49b1-a6a8-cb33818ce7ee-openshift-service-ca\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.884423 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.884398    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-metrics-certs\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.884883 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.884855    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c79098-7c03-49b1-a6a8-cb33818ce7ee-openshift-service-ca\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.886888 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.886866    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-metrics-certs\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.887079 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.887047    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-maas-api-tls\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:18.892002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:18.891973    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8cv\" (UniqueName: \"kubernetes.io/projected/86c79098-7c03-49b1-a6a8-cb33818ce7ee-kube-api-access-7k8cv\") pod \"maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:19.039795 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:19.039702    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:19.414410 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:35:19.408489    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c79098_7c03_49b1_a6a8_cb33818ce7ee.slice/crio-2f5b1a5e1978bff925f7b29269515b0cfdb7dd14d67eeb7b4e4883ec35ca98cc WatchSource:0}: Error finding container 2f5b1a5e1978bff925f7b29269515b0cfdb7dd14d67eeb7b4e4883ec35ca98cc: Status 404 returned error can't find the container with id 2f5b1a5e1978bff925f7b29269515b0cfdb7dd14d67eeb7b4e4883ec35ca98cc
Nov 03 13:35:19.415960 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:19.415925    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"]
Nov 03 13:35:19.745834 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:19.745785    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj" event={"ID":"86c79098-7c03-49b1-a6a8-cb33818ce7ee","Type":"ContainerStarted","Data":"2f5b1a5e1978bff925f7b29269515b0cfdb7dd14d67eeb7b4e4883ec35ca98cc"}
Nov 03 13:35:20.754383 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:20.754330    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj" event={"ID":"86c79098-7c03-49b1-a6a8-cb33818ce7ee","Type":"ContainerStarted","Data":"76beb0b8fd8466570627ff0e891d5142d7649ff96a62ff1cc9bb094bb0080f1e"}
Nov 03 13:35:20.754934 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:20.754451    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:20.773808 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:20.773746    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj" podStartSLOduration=2.376730157 podStartE2EDuration="2.773729049s" totalImagesPullingTime="396.998892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:20.770384632 +0000 UTC m=+1608.582052844" watchObservedRunningTime="2026-11-03 13:35:20.773729049 +0000 UTC m=+1608.585397239"
Nov 03 13:35:23.846949 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:23.846904    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-9856f4"]
Nov 03 13:35:23.852356 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:23.852321    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-9856f4"
Nov 03 13:35:23.856848 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:23.856825    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncr7v\""
Nov 03 13:35:23.857985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:23.857959    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-9856f4"]
Nov 03 13:35:23.897685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:23.897649    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw54s\" (UniqueName: \"kubernetes.io/projected/d6aff6a9-f306-4eb0-a59a-911ee999d6ba-kube-api-access-dw54s\") pod \"test-curl-7095-9856f4\" (UID: \"d6aff6a9-f306-4eb0-a59a-911ee999d6ba\") " pod="opendatahub/test-curl-7095-9856f4"
Nov 03 13:35:23.998869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:23.998831    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw54s\" (UniqueName: \"kubernetes.io/projected/d6aff6a9-f306-4eb0-a59a-911ee999d6ba-kube-api-access-dw54s\") pod \"test-curl-7095-9856f4\" (UID: \"d6aff6a9-f306-4eb0-a59a-911ee999d6ba\") " pod="opendatahub/test-curl-7095-9856f4"
Nov 03 13:35:24.007457 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:24.007420    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw54s\" (UniqueName: \"kubernetes.io/projected/d6aff6a9-f306-4eb0-a59a-911ee999d6ba-kube-api-access-dw54s\") pod \"test-curl-7095-9856f4\" (UID: \"d6aff6a9-f306-4eb0-a59a-911ee999d6ba\") " pod="opendatahub/test-curl-7095-9856f4"
Nov 03 13:35:24.171370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:24.171252    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-9856f4"
Nov 03 13:35:24.455182 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:24.455151    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-9856f4"]
Nov 03 13:35:24.456456 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:35:24.456427    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6aff6a9_f306_4eb0_a59a_911ee999d6ba.slice/crio-59536ec9bb6340cbeddd4e49906679abd1699eafac270a73ffde5a34b0e231a6 WatchSource:0}: Error finding container 59536ec9bb6340cbeddd4e49906679abd1699eafac270a73ffde5a34b0e231a6: Status 404 returned error can't find the container with id 59536ec9bb6340cbeddd4e49906679abd1699eafac270a73ffde5a34b0e231a6
Nov 03 13:35:24.783772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:24.783684    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-9856f4" event={"ID":"d6aff6a9-f306-4eb0-a59a-911ee999d6ba","Type":"ContainerStarted","Data":"d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951"}
Nov 03 13:35:24.783772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:24.783731    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-9856f4" event={"ID":"d6aff6a9-f306-4eb0-a59a-911ee999d6ba","Type":"ContainerStarted","Data":"59536ec9bb6340cbeddd4e49906679abd1699eafac270a73ffde5a34b0e231a6"}
Nov 03 13:35:24.801096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:24.801032    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7095-9856f4" podStartSLOduration=1.8010180660000001 podStartE2EDuration="1.801018066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:24.800178519 +0000 UTC m=+1612.611846729" watchObservedRunningTime="2026-11-03 13:35:24.801018066 +0000 UTC m=+1612.612686253"
Nov 03 13:35:25.488232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.488199    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-9856f4"]
Nov 03 13:35:25.498393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.498202    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-9856f4"]
Nov 03 13:35:25.651059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.651024    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-d9dfdd"]
Nov 03 13:35:25.651879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.651854    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d6aff6a9-f306-4eb0-a59a-911ee999d6ba" containerName="test-curl-7095-9856f4"
Nov 03 13:35:25.651879 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.651876    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6aff6a9-f306-4eb0-a59a-911ee999d6ba" containerName="test-curl-7095-9856f4"
Nov 03 13:35:25.652062 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.651980    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6aff6a9-f306-4eb0-a59a-911ee999d6ba" containerName="test-curl-7095-9856f4"
Nov 03 13:35:25.655969 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.655947    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d9dfdd"
Nov 03 13:35:25.659662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.659641    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-d9dfdd"]
Nov 03 13:35:25.722581 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.722541    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5km82\" (UniqueName: \"kubernetes.io/projected/efdd2a45-8a1e-44bd-84e8-49ac68556a53-kube-api-access-5km82\") pod \"test-curl-7095-d9dfdd\" (UID: \"efdd2a45-8a1e-44bd-84e8-49ac68556a53\") " pod="opendatahub/test-curl-7095-d9dfdd"
Nov 03 13:35:25.823852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.823765    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5km82\" (UniqueName: \"kubernetes.io/projected/efdd2a45-8a1e-44bd-84e8-49ac68556a53-kube-api-access-5km82\") pod \"test-curl-7095-d9dfdd\" (UID: \"efdd2a45-8a1e-44bd-84e8-49ac68556a53\") " pod="opendatahub/test-curl-7095-d9dfdd"
Nov 03 13:35:25.833474 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.833446    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5km82\" (UniqueName: \"kubernetes.io/projected/efdd2a45-8a1e-44bd-84e8-49ac68556a53-kube-api-access-5km82\") pod \"test-curl-7095-d9dfdd\" (UID: \"efdd2a45-8a1e-44bd-84e8-49ac68556a53\") " pod="opendatahub/test-curl-7095-d9dfdd"
Nov 03 13:35:25.974037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:25.974000    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d9dfdd"
Nov 03 13:35:26.271566 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:26.271534    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-d9dfdd"]
Nov 03 13:35:26.767381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:26.767322    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:35:26.802391 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:26.801054    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-d9dfdd" event={"ID":"efdd2a45-8a1e-44bd-84e8-49ac68556a53","Type":"ContainerStarted","Data":"d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8"}
Nov 03 13:35:26.802391 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:26.801154    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-d9dfdd" event={"ID":"efdd2a45-8a1e-44bd-84e8-49ac68556a53","Type":"ContainerStarted","Data":"4b81e9908ecacc0b2145e629b572fa304ad8157f35ba7d90a46ebd51cb78a850"}
Nov 03 13:35:26.802391 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:26.801280    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7095-9856f4" podUID="d6aff6a9-f306-4eb0-a59a-911ee999d6ba" containerName="test-curl-7095-9856f4" containerID="cri-o://d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951" gracePeriod=2
Nov 03 13:35:27.649135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.649078    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7095-d9dfdd" podStartSLOduration=2.649052335 podStartE2EDuration="2.649052335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:26.822617791 +0000 UTC m=+1614.634285983" watchObservedRunningTime="2026-11-03 13:35:27.649052335 +0000 UTC m=+1615.460720523"
Nov 03 13:35:27.651725 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.651695    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-d9dfdd"]
Nov 03 13:35:27.662059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.660541    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-d9dfdd"]
Nov 03 13:35:27.815913 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.815863    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-f2afce"]
Nov 03 13:35:27.817815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.817788    4447 cpu_manager.go:414] "containerMap: removing container" podUID="efdd2a45-8a1e-44bd-84e8-49ac68556a53" containerName="test-curl-7095-d9dfdd"
Nov 03 13:35:27.818053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.818034    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efdd2a45-8a1e-44bd-84e8-49ac68556a53" containerName="test-curl-7095-d9dfdd"
Nov 03 13:35:27.818566 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.818545    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="efdd2a45-8a1e-44bd-84e8-49ac68556a53" containerName="test-curl-7095-d9dfdd"
Nov 03 13:35:27.824164 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.824132    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-f2afce"
Nov 03 13:35:27.829511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.829481    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-f2afce"]
Nov 03 13:35:27.882460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.882396    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4c7t\" (UniqueName: \"kubernetes.io/projected/37c5d36a-0dc4-4fce-81f2-6f16e8519f2d-kube-api-access-f4c7t\") pod \"test-curl-7095-f2afce\" (UID: \"37c5d36a-0dc4-4fce-81f2-6f16e8519f2d\") " pod="opendatahub/test-curl-7095-f2afce"
Nov 03 13:35:27.984069 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.984032    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4c7t\" (UniqueName: \"kubernetes.io/projected/37c5d36a-0dc4-4fce-81f2-6f16e8519f2d-kube-api-access-f4c7t\") pod \"test-curl-7095-f2afce\" (UID: \"37c5d36a-0dc4-4fce-81f2-6f16e8519f2d\") " pod="opendatahub/test-curl-7095-f2afce"
Nov 03 13:35:27.996536 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:27.996499    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4c7t\" (UniqueName: \"kubernetes.io/projected/37c5d36a-0dc4-4fce-81f2-6f16e8519f2d-kube-api-access-f4c7t\") pod \"test-curl-7095-f2afce\" (UID: \"37c5d36a-0dc4-4fce-81f2-6f16e8519f2d\") " pod="opendatahub/test-curl-7095-f2afce"
Nov 03 13:35:28.150379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:28.150352    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-f2afce"
Nov 03 13:35:28.437151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:28.437122    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-f2afce"]
Nov 03 13:35:28.438385 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:35:28.438338    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37c5d36a_0dc4_4fce_81f2_6f16e8519f2d.slice/crio-6f585a359c9a0e67464d3986c1101b7fe1414e5aa5afdfd3aca8e369323a3a79 WatchSource:0}: Error finding container 6f585a359c9a0e67464d3986c1101b7fe1414e5aa5afdfd3aca8e369323a3a79: Status 404 returned error can't find the container with id 6f585a359c9a0e67464d3986c1101b7fe1414e5aa5afdfd3aca8e369323a3a79
Nov 03 13:35:28.833691 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:28.833650    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7095-d9dfdd" podUID="efdd2a45-8a1e-44bd-84e8-49ac68556a53" containerName="test-curl-7095-d9dfdd" containerID="cri-o://d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8" gracePeriod=2
Nov 03 13:35:28.834279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:28.833961    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-f2afce" event={"ID":"37c5d36a-0dc4-4fce-81f2-6f16e8519f2d","Type":"ContainerStarted","Data":"92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b"}
Nov 03 13:35:28.834279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:28.833997    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-f2afce" event={"ID":"37c5d36a-0dc4-4fce-81f2-6f16e8519f2d","Type":"ContainerStarted","Data":"6f585a359c9a0e67464d3986c1101b7fe1414e5aa5afdfd3aca8e369323a3a79"}
Nov 03 13:35:28.852323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:28.852268    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7095-f2afce" podStartSLOduration=1.8522504899999999 podStartE2EDuration="1.85225049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:28.847257437 +0000 UTC m=+1616.658925645" watchObservedRunningTime="2026-11-03 13:35:28.85225049 +0000 UTC m=+1616.663918679"
Nov 03 13:35:29.000358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.000316    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-9856f4"
Nov 03 13:35:29.102011 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.101927    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6aff6a9-f306-4eb0-a59a-911ee999d6ba-kube-api-access-dw54s\" (UniqueName: \"kubernetes.io/projected/d6aff6a9-f306-4eb0-a59a-911ee999d6ba-kube-api-access-dw54s\") pod \"d6aff6a9-f306-4eb0-a59a-911ee999d6ba\" (UID: \"d6aff6a9-f306-4eb0-a59a-911ee999d6ba\") "
Nov 03 13:35:29.104035 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.104002    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6aff6a9-f306-4eb0-a59a-911ee999d6ba-kube-api-access-dw54s" pod "d6aff6a9-f306-4eb0-a59a-911ee999d6ba" (UID: "d6aff6a9-f306-4eb0-a59a-911ee999d6ba"). InnerVolumeSpecName "kube-api-access-dw54s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:29.203215 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.203180    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw54s\" (UniqueName: \"kubernetes.io/projected/d6aff6a9-f306-4eb0-a59a-911ee999d6ba-kube-api-access-dw54s\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:29.522754 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.522719    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-f2afce"]
Nov 03 13:35:29.527991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.527963    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-f2afce"]
Nov 03 13:35:29.675559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.675525    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-11eecc"]
Nov 03 13:35:29.676186 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.676173    4447 cpu_manager.go:414] "containerMap: removing container" podUID="37c5d36a-0dc4-4fce-81f2-6f16e8519f2d" containerName="test-curl-7095-f2afce"
Nov 03 13:35:29.676237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.676187    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c5d36a-0dc4-4fce-81f2-6f16e8519f2d" containerName="test-curl-7095-f2afce"
Nov 03 13:35:29.676320 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.676311    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c5d36a-0dc4-4fce-81f2-6f16e8519f2d" containerName="test-curl-7095-f2afce"
Nov 03 13:35:29.679875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.679856    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-11eecc"
Nov 03 13:35:29.687950 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.687927    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-11eecc"]
Nov 03 13:35:29.811305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.811224    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8dm\" (UniqueName: \"kubernetes.io/projected/0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea-kube-api-access-pm8dm\") pod \"test-curl-7095-11eecc\" (UID: \"0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea\") " pod="opendatahub/test-curl-7095-11eecc"
Nov 03 13:35:29.841716 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.841677    4447 generic.go:356] "Generic (PLEG): container finished" podID="d6aff6a9-f306-4eb0-a59a-911ee999d6ba" containerID="d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951" exitCode=137
Nov 03 13:35:29.842227 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.841861    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-9856f4"
Nov 03 13:35:29.842492 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.842331    4447 scope.go:122] "RemoveContainer" containerID="d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951"
Nov 03 13:35:29.853610 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.853593    4447 scope.go:122] "RemoveContainer" containerID="d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951"
Nov 03 13:35:29.853761 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:29.853736    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-9856f4\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="d6aff6a9-f306-4eb0-a59a-911ee999d6ba" pod="opendatahub/test-curl-7095-9856f4"
Nov 03 13:35:29.853874 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:29.853853    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951\": container with ID starting with d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951 not found: ID does not exist" containerID="d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951"
Nov 03 13:35:29.853911 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.853882    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951"} err="failed to get container status \"d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951\": rpc error: code = NotFound desc = could not find container \"d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951\": container with ID starting with d559ab868eb52b5d8c09a534f54ba14e99c6ae8c8fbf26268e11b56790b91951 not found: ID does not exist"
Nov 03 13:35:29.913106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.913068    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8dm\" (UniqueName: \"kubernetes.io/projected/0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea-kube-api-access-pm8dm\") pod \"test-curl-7095-11eecc\" (UID: \"0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea\") " pod="opendatahub/test-curl-7095-11eecc"
Nov 03 13:35:29.922314 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.922284    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8dm\" (UniqueName: \"kubernetes.io/projected/0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea-kube-api-access-pm8dm\") pod \"test-curl-7095-11eecc\" (UID: \"0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea\") " pod="opendatahub/test-curl-7095-11eecc"
Nov 03 13:35:29.996299 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:29.996264    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-11eecc"
Nov 03 13:35:30.290005 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:30.289771    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-11eecc"]
Nov 03 13:35:30.343158 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:30.343125    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6aff6a9-f306-4eb0-a59a-911ee999d6ba" path="/var/lib/kubelet/pods/d6aff6a9-f306-4eb0-a59a-911ee999d6ba/volumes"
Nov 03 13:35:30.865669 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:30.865624    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-11eecc" event={"ID":"0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea","Type":"ContainerStarted","Data":"048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93"}
Nov 03 13:35:30.866116 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:30.865688    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-11eecc" event={"ID":"0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea","Type":"ContainerStarted","Data":"26ec7d6fed6b46890662ce74c8dad4a9724b8600cdc485b485603afea52d38de"}
Nov 03 13:35:30.866116 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:30.865832    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7095-f2afce" podUID="37c5d36a-0dc4-4fce-81f2-6f16e8519f2d" containerName="test-curl-7095-f2afce" containerID="cri-o://92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b" gracePeriod=2
Nov 03 13:35:30.889633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:30.889582    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7095-11eecc" podStartSLOduration=1.889568388 podStartE2EDuration="1.889568388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:30.884675009 +0000 UTC m=+1618.696343197" watchObservedRunningTime="2026-11-03 13:35:30.889568388 +0000 UTC m=+1618.701236575"
Nov 03 13:35:31.093570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:31.093538    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d9dfdd"
Nov 03 13:35:31.232366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:31.231743    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efdd2a45-8a1e-44bd-84e8-49ac68556a53-kube-api-access-5km82\" (UniqueName: \"kubernetes.io/projected/efdd2a45-8a1e-44bd-84e8-49ac68556a53-kube-api-access-5km82\") pod \"efdd2a45-8a1e-44bd-84e8-49ac68556a53\" (UID: \"efdd2a45-8a1e-44bd-84e8-49ac68556a53\") "
Nov 03 13:35:31.263363 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:31.258911    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdd2a45-8a1e-44bd-84e8-49ac68556a53-kube-api-access-5km82" pod "efdd2a45-8a1e-44bd-84e8-49ac68556a53" (UID: "efdd2a45-8a1e-44bd-84e8-49ac68556a53"). InnerVolumeSpecName "kube-api-access-5km82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:31.333692 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:31.333658    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5km82\" (UniqueName: \"kubernetes.io/projected/efdd2a45-8a1e-44bd-84e8-49ac68556a53-kube-api-access-5km82\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:31.930842 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:31.930806    4447 generic.go:356] "Generic (PLEG): container finished" podID="efdd2a45-8a1e-44bd-84e8-49ac68556a53" containerID="d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8" exitCode=137
Nov 03 13:35:31.932403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:31.932380    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d9dfdd"
Nov 03 13:35:31.942532 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:31.942503    4447 scope.go:122] "RemoveContainer" containerID="d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8"
Nov 03 13:35:31.985611 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:31.985579    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-11eecc"]
Nov 03 13:35:32.144366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:32.120903    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-11eecc"]
Nov 03 13:35:32.329367 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:32.289478    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-d9dfdd\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="efdd2a45-8a1e-44bd-84e8-49ac68556a53" pod="opendatahub/test-curl-7095-d9dfdd"
Nov 03 13:35:32.333071 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:32.332983    4447 scope.go:122] "RemoveContainer" containerID="d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8"
Nov 03 13:35:32.333474 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:32.333442    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8\": container with ID starting with d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8 not found: ID does not exist" containerID="d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8"
Nov 03 13:35:32.333571 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:32.333486    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8"} err="failed to get container status \"d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8\": rpc error: code = NotFound desc = could not find container \"d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8\": container with ID starting with d1b1fe56ff0b077677c2c396b4396c5b8d856f11b7f17841c9c6d149f16c77b8 not found: ID does not exist"
Nov 03 13:35:32.405566 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:32.401197    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-d9dfdd\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="efdd2a45-8a1e-44bd-84e8-49ac68556a53" pod="opendatahub/test-curl-7095-d9dfdd"
Nov 03 13:35:32.448366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:32.444800    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efdd2a45-8a1e-44bd-84e8-49ac68556a53" path="/var/lib/kubelet/pods/efdd2a45-8a1e-44bd-84e8-49ac68556a53/volumes"
Nov 03 13:35:33.011830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:33.011787    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7095-11eecc" podUID="0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea" containerName="test-curl-7095-11eecc" containerID="cri-o://048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93" gracePeriod=2
Nov 03 13:35:33.641846 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:33.641790    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:35:33.851404 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:33.851369    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-f2afce"
Nov 03 13:35:33.906666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:33.906582    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c5d36a-0dc4-4fce-81f2-6f16e8519f2d-kube-api-access-f4c7t\" (UniqueName: \"kubernetes.io/projected/37c5d36a-0dc4-4fce-81f2-6f16e8519f2d-kube-api-access-f4c7t\") pod \"37c5d36a-0dc4-4fce-81f2-6f16e8519f2d\" (UID: \"37c5d36a-0dc4-4fce-81f2-6f16e8519f2d\") "
Nov 03 13:35:33.910082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:33.910019    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c5d36a-0dc4-4fce-81f2-6f16e8519f2d-kube-api-access-f4c7t" pod "37c5d36a-0dc4-4fce-81f2-6f16e8519f2d" (UID: "37c5d36a-0dc4-4fce-81f2-6f16e8519f2d"). InnerVolumeSpecName "kube-api-access-f4c7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:34.009428 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:34.009392    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4c7t\" (UniqueName: \"kubernetes.io/projected/37c5d36a-0dc4-4fce-81f2-6f16e8519f2d-kube-api-access-f4c7t\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:34.023084 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:34.020955    4447 generic.go:356] "Generic (PLEG): container finished" podID="37c5d36a-0dc4-4fce-81f2-6f16e8519f2d" containerID="92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b" exitCode=137
Nov 03 13:35:34.023084 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:34.021114    4447 scope.go:122] "RemoveContainer" containerID="92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b"
Nov 03 13:35:34.023084 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:34.021230    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-f2afce"
Nov 03 13:35:34.032649 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:34.032625    4447 scope.go:122] "RemoveContainer" containerID="92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b"
Nov 03 13:35:34.033002 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:34.032980    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b\": container with ID starting with 92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b not found: ID does not exist" containerID="92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b"
Nov 03 13:35:34.033097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:34.033013    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b"} err="failed to get container status \"92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b\": rpc error: code = NotFound desc = could not find container \"92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b\": container with ID starting with 92dd6a5c5fc000f238a57e8145ba3f3822920f09d539e45d6bbe3dabcdd1968b not found: ID does not exist"
Nov 03 13:35:34.037889 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:34.037862    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-f2afce\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="37c5d36a-0dc4-4fce-81f2-6f16e8519f2d" pod="opendatahub/test-curl-7095-f2afce"
Nov 03 13:35:34.321441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:34.321409    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c5d36a-0dc4-4fce-81f2-6f16e8519f2d" path="/var/lib/kubelet/pods/37c5d36a-0dc4-4fce-81f2-6f16e8519f2d/volumes"
Nov 03 13:35:35.290653 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:35.290628    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-11eecc"
Nov 03 13:35:35.330574 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:35.330492    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea-kube-api-access-pm8dm\" (UniqueName: \"kubernetes.io/projected/0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea-kube-api-access-pm8dm\") pod \"0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea\" (UID: \"0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea\") "
Nov 03 13:35:35.332570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:35.332537    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea-kube-api-access-pm8dm" pod "0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea" (UID: "0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea"). InnerVolumeSpecName "kube-api-access-pm8dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:35.432570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:35.432531    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm8dm\" (UniqueName: \"kubernetes.io/projected/0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea-kube-api-access-pm8dm\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:36.038789 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.038750    4447 generic.go:356] "Generic (PLEG): container finished" podID="0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea" containerID="048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93" exitCode=137
Nov 03 13:35:36.038983 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.038795    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-11eecc"
Nov 03 13:35:36.038983 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.038858    4447 scope.go:122] "RemoveContainer" containerID="048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93"
Nov 03 13:35:36.048889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.048872    4447 scope.go:122] "RemoveContainer" containerID="048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93"
Nov 03 13:35:36.049171 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:36.049144    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93\": container with ID starting with 048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93 not found: ID does not exist" containerID="048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93"
Nov 03 13:35:36.049239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.049182    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93"} err="failed to get container status \"048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93\": rpc error: code = NotFound desc = could not find container \"048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93\": container with ID starting with 048a9f15554eccb2036da62c6671538871cd30b530e68029f593a6286854dc93 not found: ID does not exist"
Nov 03 13:35:36.049907 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:36.049887    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-11eecc\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea" pod="opendatahub/test-curl-7095-11eecc"
Nov 03 13:35:36.331244 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.331169    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea" path="/var/lib/kubelet/pods/0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea/volumes"
Nov 03 13:35:36.439785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.439755    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"]
Nov 03 13:35:36.440495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.440475    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea" containerName="test-curl-7095-11eecc"
Nov 03 13:35:36.440495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.440493    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea" containerName="test-curl-7095-11eecc"
Nov 03 13:35:36.440680 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.440644    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fab472a-a6c0-493d-ba14-4e8f7fdbb9ea" containerName="test-curl-7095-11eecc"
Nov 03 13:35:36.446122 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.446098    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.456393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.450073    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c47670-serving-cert\""
Nov 03 13:35:36.456393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.453016    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"]
Nov 03 13:35:36.558978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.556357    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-metrics-certs\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.558978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.556479    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.558978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.556574    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da579797-607a-4132-945d-69c919276f0d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.558978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.556622    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8sn\" (UniqueName: \"kubernetes.io/projected/da579797-607a-4132-945d-69c919276f0d-kube-api-access-tf8sn\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.657770 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.657687    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-metrics-certs\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.657770 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.657768    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.658064 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.657817    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da579797-607a-4132-945d-69c919276f0d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.658064 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.657862    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8sn\" (UniqueName: \"kubernetes.io/projected/da579797-607a-4132-945d-69c919276f0d-kube-api-access-tf8sn\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.658064 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:36.657929    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-serving-cert: secret "maas-api-e2e-extmod-a-c47670-serving-cert" not found
Nov 03 13:35:36.658064 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:36.658000    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls podName:da579797-607a-4132-945d-69c919276f0d nodeName:}" failed. No retries permitted until 2026-11-03 13:35:37.157975264 +0000 UTC m=+1624.969643445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls") pod "maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx" (UID: "da579797-607a-4132-945d-69c919276f0d") : secret "maas-api-e2e-extmod-a-c47670-serving-cert" not found
Nov 03 13:35:36.658602 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.658579    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da579797-607a-4132-945d-69c919276f0d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.660414 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.660389    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-metrics-certs\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:36.667385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:36.667324    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8sn\" (UniqueName: \"kubernetes.io/projected/da579797-607a-4132-945d-69c919276f0d-kube-api-access-tf8sn\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:37.164648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:37.164590    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:37.168047 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:37.168013    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:37.365369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:37.364186    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:37.692096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:37.692074    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"]
Nov 03 13:35:37.694915 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:35:37.694885    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda579797_607a_4132_945d_69c919276f0d.slice/crio-d33b181259245f935100407c29f3a79113270b836ac6e5060708ac14e50ffd8a WatchSource:0}: Error finding container d33b181259245f935100407c29f3a79113270b836ac6e5060708ac14e50ffd8a: Status 404 returned error can't find the container with id d33b181259245f935100407c29f3a79113270b836ac6e5060708ac14e50ffd8a
Nov 03 13:35:38.057275 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:38.057171    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx" event={"ID":"da579797-607a-4132-945d-69c919276f0d","Type":"ContainerStarted","Data":"d33b181259245f935100407c29f3a79113270b836ac6e5060708ac14e50ffd8a"}
Nov 03 13:35:39.067667 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:39.067625    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx" event={"ID":"da579797-607a-4132-945d-69c919276f0d","Type":"ContainerStarted","Data":"b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f"}
Nov 03 13:35:39.068116 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:39.067760    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:40.932322 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:40.932255    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx" podStartSLOduration=4.518000359 podStartE2EDuration="4.932234228s" totalImagesPullingTime="414.233869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:39.095647041 +0000 UTC m=+1626.907315241" watchObservedRunningTime="2026-11-03 13:35:40.932234228 +0000 UTC m=+1628.743902417"
Nov 03 13:35:40.934605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:40.934577    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-32cedd"]
Nov 03 13:35:40.940285 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:40.940260    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-32cedd"
Nov 03 13:35:40.944810 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:40.944723    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncr7v\""
Nov 03 13:35:40.944934 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:40.944861    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-32cedd"]
Nov 03 13:35:41.042698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:41.042649    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg8mt\" (UniqueName: \"kubernetes.io/projected/deb245c5-e749-4d1c-8695-fec8cdb642e4-kube-api-access-qg8mt\") pod \"test-curl-7092-32cedd\" (UID: \"deb245c5-e749-4d1c-8695-fec8cdb642e4\") " pod="opendatahub/test-curl-7092-32cedd"
Nov 03 13:35:41.144602 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:41.144564    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg8mt\" (UniqueName: \"kubernetes.io/projected/deb245c5-e749-4d1c-8695-fec8cdb642e4-kube-api-access-qg8mt\") pod \"test-curl-7092-32cedd\" (UID: \"deb245c5-e749-4d1c-8695-fec8cdb642e4\") " pod="opendatahub/test-curl-7092-32cedd"
Nov 03 13:35:41.152042 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:41.152013    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg8mt\" (UniqueName: \"kubernetes.io/projected/deb245c5-e749-4d1c-8695-fec8cdb642e4-kube-api-access-qg8mt\") pod \"test-curl-7092-32cedd\" (UID: \"deb245c5-e749-4d1c-8695-fec8cdb642e4\") " pod="opendatahub/test-curl-7092-32cedd"
Nov 03 13:35:41.259265 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:41.259168    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-32cedd"
Nov 03 13:35:41.556986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:41.556959    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-32cedd"]
Nov 03 13:35:41.559283 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:35:41.559252    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeb245c5_e749_4d1c_8695_fec8cdb642e4.slice/crio-9820124fe85e2ef8622a0aaf9c83642b09045ffc4e25132dd1183188436f4813 WatchSource:0}: Error finding container 9820124fe85e2ef8622a0aaf9c83642b09045ffc4e25132dd1183188436f4813: Status 404 returned error can't find the container with id 9820124fe85e2ef8622a0aaf9c83642b09045ffc4e25132dd1183188436f4813
Nov 03 13:35:42.094396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:42.094362    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-32cedd" event={"ID":"deb245c5-e749-4d1c-8695-fec8cdb642e4","Type":"ContainerStarted","Data":"e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a"}
Nov 03 13:35:42.094396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:42.094403    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-32cedd" event={"ID":"deb245c5-e749-4d1c-8695-fec8cdb642e4","Type":"ContainerStarted","Data":"9820124fe85e2ef8622a0aaf9c83642b09045ffc4e25132dd1183188436f4813"}
Nov 03 13:35:42.109105 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:42.109042    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7092-32cedd" podStartSLOduration=2.10902729 podStartE2EDuration="2.10902729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:42.106313413 +0000 UTC m=+1629.917981604" watchObservedRunningTime="2026-11-03 13:35:42.10902729 +0000 UTC m=+1629.920695478"
Nov 03 13:35:42.824356 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:42.824303    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-32cedd"]
Nov 03 13:35:42.833131 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:42.833093    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-32cedd"]
Nov 03 13:35:44.116872 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:44.116830    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7092-32cedd" podUID="deb245c5-e749-4d1c-8695-fec8cdb642e4" containerName="test-curl-7092-32cedd" containerID="cri-o://e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a" gracePeriod=2
Nov 03 13:35:44.120579 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:44.120540    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-32cedd\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="deb245c5-e749-4d1c-8695-fec8cdb642e4" pod="opendatahub/test-curl-7092-32cedd"
Nov 03 13:35:45.080938 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:45.080908    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:35:46.384579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:46.384520    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-32cedd"
Nov 03 13:35:46.538620 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:46.538575    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb245c5-e749-4d1c-8695-fec8cdb642e4-kube-api-access-qg8mt\" (UniqueName: \"kubernetes.io/projected/deb245c5-e749-4d1c-8695-fec8cdb642e4-kube-api-access-qg8mt\") pod \"deb245c5-e749-4d1c-8695-fec8cdb642e4\" (UID: \"deb245c5-e749-4d1c-8695-fec8cdb642e4\") "
Nov 03 13:35:46.540642 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:46.540617    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb245c5-e749-4d1c-8695-fec8cdb642e4-kube-api-access-qg8mt" pod "deb245c5-e749-4d1c-8695-fec8cdb642e4" (UID: "deb245c5-e749-4d1c-8695-fec8cdb642e4"). InnerVolumeSpecName "kube-api-access-qg8mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:46.640395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:46.640301    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg8mt\" (UniqueName: \"kubernetes.io/projected/deb245c5-e749-4d1c-8695-fec8cdb642e4-kube-api-access-qg8mt\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:47.152430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:47.152396    4447 generic.go:356] "Generic (PLEG): container finished" podID="deb245c5-e749-4d1c-8695-fec8cdb642e4" containerID="e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a" exitCode=137
Nov 03 13:35:47.152641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:47.152462    4447 scope.go:122] "RemoveContainer" containerID="e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a"
Nov 03 13:35:47.152641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:47.152460    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-32cedd"
Nov 03 13:35:47.162752 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:47.162731    4447 scope.go:122] "RemoveContainer" containerID="e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a"
Nov 03 13:35:47.163076 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:47.163049    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a\": container with ID starting with e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a not found: ID does not exist" containerID="e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a"
Nov 03 13:35:47.163159 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:47.163089    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a"} err="failed to get container status \"e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a\": rpc error: code = NotFound desc = could not find container \"e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a\": container with ID starting with e5e5472786ee16a7130ad9a0e0e24652aae5491c499f33024b4fc3ae9616b78a not found: ID does not exist"
Nov 03 13:35:47.166531 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:35:47.166499    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7092-32cedd\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="deb245c5-e749-4d1c-8695-fec8cdb642e4" pod="opendatahub/test-curl-7092-32cedd"
Nov 03 13:35:48.321287 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:48.321256    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb245c5-e749-4d1c-8695-fec8cdb642e4" path="/var/lib/kubelet/pods/deb245c5-e749-4d1c-8695-fec8cdb642e4/volumes"
Nov 03 13:35:49.767072 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.767028    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"]
Nov 03 13:35:49.768123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.768093    4447 cpu_manager.go:414] "containerMap: removing container" podUID="deb245c5-e749-4d1c-8695-fec8cdb642e4" containerName="test-curl-7092-32cedd"
Nov 03 13:35:49.768123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.768121    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb245c5-e749-4d1c-8695-fec8cdb642e4" containerName="test-curl-7092-32cedd"
Nov 03 13:35:49.768393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.768335    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb245c5-e749-4d1c-8695-fec8cdb642e4" containerName="test-curl-7092-32cedd"
Nov 03 13:35:49.774701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.774632    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.789687 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.789648    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2b8a08\"/\"sub-test-model-2b8a08-kserve-self-signed-certs\""
Nov 03 13:35:49.789867 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.789702    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2b8a08\"/\"openshift-service-ca.crt\""
Nov 03 13:35:49.789867 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.789745    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2b8a08\"/\"default-dockercfg-r8st8\""
Nov 03 13:35:49.789867 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.789845    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2b8a08\"/\"kube-root-ca.crt\""
Nov 03 13:35:49.790039 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.789973    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"]
Nov 03 13:35:49.790751 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.790725    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-tmp-dir\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.790958 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.790938    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-dshm\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.791226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.791202    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-model-cache\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.792525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.792498    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d05f68-2b0a-4410-a346-793bab7aabae-tls-certs\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.792647 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.792566    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkz9x\" (UniqueName: \"kubernetes.io/projected/a1d05f68-2b0a-4410-a346-793bab7aabae-kube-api-access-vkz9x\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.792713 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.792667    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-home\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.894239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.894194    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-tmp-dir\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.894432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.894262    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-dshm\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.894729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.894472    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-model-cache\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.894729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.894553    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d05f68-2b0a-4410-a346-793bab7aabae-tls-certs\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.894729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.894584    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkz9x\" (UniqueName: \"kubernetes.io/projected/a1d05f68-2b0a-4410-a346-793bab7aabae-kube-api-access-vkz9x\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.894729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.894646    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-home\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.894729 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.894683    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-tmp-dir\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.895024 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.894767    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-model-cache\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.895071 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.895017    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-home\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.896915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.896892    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-dshm\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.897062 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.897034    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d05f68-2b0a-4410-a346-793bab7aabae-tls-certs\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:49.911007 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:49.910972    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkz9x\" (UniqueName: \"kubernetes.io/projected/a1d05f68-2b0a-4410-a346-793bab7aabae-kube-api-access-vkz9x\") pod \"sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") " pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:50.104514 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:50.104419    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:35:50.449389 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:50.449360    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"]
Nov 03 13:35:50.451050 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:35:50.451021    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1d05f68_2b0a_4410_a346_793bab7aabae.slice/crio-91f171c593c7e7381bcc55661207354b84c83ca613658c60a272d14d470e169e WatchSource:0}: Error finding container 91f171c593c7e7381bcc55661207354b84c83ca613658c60a272d14d470e169e: Status 404 returned error can't find the container with id 91f171c593c7e7381bcc55661207354b84c83ca613658c60a272d14d470e169e
Nov 03 13:35:51.190087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:51.190047    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4" event={"ID":"a1d05f68-2b0a-4410-a346-793bab7aabae","Type":"ContainerStarted","Data":"6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2"}
Nov 03 13:35:51.190568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:51.190102    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4" event={"ID":"a1d05f68-2b0a-4410-a346-793bab7aabae","Type":"ContainerStarted","Data":"91f171c593c7e7381bcc55661207354b84c83ca613658c60a272d14d470e169e"}
Nov 03 13:35:51.210964 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:51.210910    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4" podStartSLOduration=2.210895822 podStartE2EDuration="2.210895822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:51.209627853 +0000 UTC m=+1639.021296064" watchObservedRunningTime="2026-11-03 13:35:51.210895822 +0000 UTC m=+1639.022564015"
Nov 03 13:35:52.105128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:35:52.105095    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:36:00.105058 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:00.105016    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:36:00.119133 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:00.119101    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:36:00.322776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:00.322740    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:36:03.415083 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:36:03.415043    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:36:03.935399 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:03.935356    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"]
Nov 03 13:36:03.935668 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:03.935617    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy" containerID="cri-o://e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7" gracePeriod=30
Nov 03 13:36:09.217231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.217175    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:36:09.333684 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.333653    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-certs\") pod \"6394bce3-b0a3-4f68-af08-0f0fff106087\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") "
Nov 03 13:36:09.333886 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.333698    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-kube-api-access-vtzhl\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-kube-api-access-vtzhl\") pod \"6394bce3-b0a3-4f68-af08-0f0fff106087\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") "
Nov 03 13:36:09.333886 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.333761    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-token\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-token\") pod \"6394bce3-b0a3-4f68-af08-0f0fff106087\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") "
Nov 03 13:36:09.333886 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.333785    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-envoy\") pod \"6394bce3-b0a3-4f68-af08-0f0fff106087\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") "
Nov 03 13:36:09.334052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.333904    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-podinfo\") pod \"6394bce3-b0a3-4f68-af08-0f0fff106087\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") "
Nov 03 13:36:09.334052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.333942    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-certs" pod "6394bce3-b0a3-4f68-af08-0f0fff106087" (UID: "6394bce3-b0a3-4f68-af08-0f0fff106087"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:09.334052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.333953    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-credential-socket\") pod \"6394bce3-b0a3-4f68-af08-0f0fff106087\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") "
Nov 03 13:36:09.334205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334053    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-data\") pod \"6394bce3-b0a3-4f68-af08-0f0fff106087\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") "
Nov 03 13:36:09.334205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334112    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6394bce3-b0a3-4f68-af08-0f0fff106087-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6394bce3-b0a3-4f68-af08-0f0fff106087-istiod-ca-cert\") pod \"6394bce3-b0a3-4f68-af08-0f0fff106087\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") "
Nov 03 13:36:09.334205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334146    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-credential-socket" pod "6394bce3-b0a3-4f68-af08-0f0fff106087" (UID: "6394bce3-b0a3-4f68-af08-0f0fff106087"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:09.334205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334175    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-socket\") pod \"6394bce3-b0a3-4f68-af08-0f0fff106087\" (UID: \"6394bce3-b0a3-4f68-af08-0f0fff106087\") "
Nov 03 13:36:09.334451 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334318    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-data" pod "6394bce3-b0a3-4f68-af08-0f0fff106087" (UID: "6394bce3-b0a3-4f68-af08-0f0fff106087"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:09.334545 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334522    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6394bce3-b0a3-4f68-af08-0f0fff106087-istiod-ca-cert" pod "6394bce3-b0a3-4f68-af08-0f0fff106087" (UID: "6394bce3-b0a3-4f68-af08-0f0fff106087"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:36:09.334600 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334546    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-socket" pod "6394bce3-b0a3-4f68-af08-0f0fff106087" (UID: "6394bce3-b0a3-4f68-af08-0f0fff106087"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:09.334922 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334892    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:09.335013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334927    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:09.335013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334943    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6394bce3-b0a3-4f68-af08-0f0fff106087-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:09.335013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334957    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:09.335013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.334971    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:09.336237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.336213    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-podinfo" pod "6394bce3-b0a3-4f68-af08-0f0fff106087" (UID: "6394bce3-b0a3-4f68-af08-0f0fff106087"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:36:09.336237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.336216    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-envoy" pod "6394bce3-b0a3-4f68-af08-0f0fff106087" (UID: "6394bce3-b0a3-4f68-af08-0f0fff106087"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:09.336398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.336238    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-kube-api-access-vtzhl" pod "6394bce3-b0a3-4f68-af08-0f0fff106087" (UID: "6394bce3-b0a3-4f68-af08-0f0fff106087"). InnerVolumeSpecName "kube-api-access-vtzhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:09.336398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.336276    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-token" pod "6394bce3-b0a3-4f68-af08-0f0fff106087" (UID: "6394bce3-b0a3-4f68-af08-0f0fff106087"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:09.380525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.380320    4447 generic.go:356] "Generic (PLEG): container finished" podID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerID="e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7" exitCode=0
Nov 03 13:36:09.380525 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.380414    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"
Nov 03 13:36:09.380780 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.380411    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" event={"ID":"6394bce3-b0a3-4f68-af08-0f0fff106087","Type":"ContainerDied","Data":"e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7"}
Nov 03 13:36:09.380780 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.380608    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8" event={"ID":"6394bce3-b0a3-4f68-af08-0f0fff106087","Type":"ContainerDied","Data":"e910a31fb28406823cfacb762d7e16207cf83e9353932fa43b8405ccd7150df7"}
Nov 03 13:36:09.380780 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.380643    4447 scope.go:122] "RemoveContainer" containerID="e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7"
Nov 03 13:36:09.391997 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.391976    4447 scope.go:122] "RemoveContainer" containerID="e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7"
Nov 03 13:36:09.392249 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:36:09.392232    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7\": container with ID starting with e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7 not found: ID does not exist" containerID="e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7"
Nov 03 13:36:09.392300 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.392257    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7"} err="failed to get container status \"e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7\": rpc error: code = NotFound desc = could not find container \"e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7\": container with ID starting with e878b3205df85bfd3237d62dc15532e4b094e6fb148c9ebcb9d41f89a67fbcc7 not found: ID does not exist"
Nov 03 13:36:09.406046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.406017    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"]
Nov 03 13:36:09.412031 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.412007    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-pgcf8"]
Nov 03 13:36:09.436053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.436028    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtzhl\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-kube-api-access-vtzhl\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:09.436053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.436053    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:09.436196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.436064    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:09.436196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:09.436072    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6394bce3-b0a3-4f68-af08-0f0fff106087-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:10.334254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:10.334218    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" path="/var/lib/kubelet/pods/6394bce3-b0a3-4f68-af08-0f0fff106087/volumes"
Nov 03 13:36:12.098845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.098810    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"]
Nov 03 13:36:12.099499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.099482    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy"
Nov 03 13:36:12.099499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.099498    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy"
Nov 03 13:36:12.099627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.099616    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6394bce3-b0a3-4f68-af08-0f0fff106087" containerName="istio-proxy"
Nov 03 13:36:12.104744 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.104725    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.112267 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.112242    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-9f71f2-serving-cert\""
Nov 03 13:36:12.118014 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.117989    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"]
Nov 03 13:36:12.274297 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.274258    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-metrics-certs\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.274503 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.274320    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bnc\" (UniqueName: \"kubernetes.io/projected/4b570f62-46d4-4b89-a7e1-9dc04c728f66-kube-api-access-44bnc\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.274577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.274561    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b570f62-46d4-4b89-a7e1-9dc04c728f66-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.274622 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.274598    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.375895 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.375813    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b570f62-46d4-4b89-a7e1-9dc04c728f66-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.375895 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.375857    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.376082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.375930    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-metrics-certs\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.376082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.375973    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44bnc\" (UniqueName: \"kubernetes.io/projected/4b570f62-46d4-4b89-a7e1-9dc04c728f66-kube-api-access-44bnc\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.376482 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.376456    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b570f62-46d4-4b89-a7e1-9dc04c728f66-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.378486 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.378466    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-metrics-certs\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.378636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.378616    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.384496 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.384471    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bnc\" (UniqueName: \"kubernetes.io/projected/4b570f62-46d4-4b89-a7e1-9dc04c728f66-kube-api-access-44bnc\") pod \"maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.423285 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.423260    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:12.754081 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.754045    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"]
Nov 03 13:36:12.760118 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:36:12.760070    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b570f62_46d4_4b89_a7e1_9dc04c728f66.slice/crio-6f8e19df83814db10d12e0a7773be94ed26c69faa18e5bb045c74a3a9c3cdc5c WatchSource:0}: Error finding container 6f8e19df83814db10d12e0a7773be94ed26c69faa18e5bb045c74a3a9c3cdc5c: Status 404 returned error can't find the container with id 6f8e19df83814db10d12e0a7773be94ed26c69faa18e5bb045c74a3a9c3cdc5c
Nov 03 13:36:12.761222 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:12.761196    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:36:13.424188 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:13.424106    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9" event={"ID":"4b570f62-46d4-4b89-a7e1-9dc04c728f66","Type":"ContainerStarted","Data":"95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3"}
Nov 03 13:36:13.424188 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:13.424167    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9" event={"ID":"4b570f62-46d4-4b89-a7e1-9dc04c728f66","Type":"ContainerStarted","Data":"6f8e19df83814db10d12e0a7773be94ed26c69faa18e5bb045c74a3a9c3cdc5c"}
Nov 03 13:36:13.424648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:13.424333    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:13.448628 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:13.448575    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9" podStartSLOduration=1.048670965 podStartE2EDuration="1.448559098s" totalImagesPullingTime="399.888133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:13.445201388 +0000 UTC m=+1661.256869598" watchObservedRunningTime="2026-11-03 13:36:13.448559098 +0000 UTC m=+1661.260227285"
Nov 03 13:36:19.436670 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:19.436587    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:36:28.116687 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.116624    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"]
Nov 03 13:36:28.132326 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.132293    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.137891 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.137863    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-146318-serving-cert\""
Nov 03 13:36:28.185947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.185908    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-metrics-certs\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.186203 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.186185    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-openshift-service-ca\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.186325 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.186310    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snh5s\" (UniqueName: \"kubernetes.io/projected/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-kube-api-access-snh5s\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.186634 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.186617    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.192544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.192509    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"]
Nov 03 13:36:28.288582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.288542    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-openshift-service-ca\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.288774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.288613    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snh5s\" (UniqueName: \"kubernetes.io/projected/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-kube-api-access-snh5s\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.288824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.288809    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.288907 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.288892    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-metrics-certs\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.288992 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:36:28.288970    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-146318-serving-cert: secret "maas-api-e2e-autores-146318-serving-cert" not found
Nov 03 13:36:28.289069 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:36:28.289050    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls podName:a5c3bfed-904b-4b38-ada6-3d3dd8c80c51 nodeName:}" failed. No retries permitted until 2026-11-03 13:36:28.789029363 +0000 UTC m=+1676.600697544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls") pod "maas-api-e2e-autores-146318-6b684595f9-86lmj" (UID: "a5c3bfed-904b-4b38-ada6-3d3dd8c80c51") : secret "maas-api-e2e-autores-146318-serving-cert" not found
Nov 03 13:36:28.289371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.289332    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-openshift-service-ca\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.291422 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.291403    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-metrics-certs\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.297122 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.297092    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snh5s\" (UniqueName: \"kubernetes.io/projected/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-kube-api-access-snh5s\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.795194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.795156    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:28.808212 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:28.808176    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls\") pod \"maas-api-e2e-autores-146318-6b684595f9-86lmj\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:29.089484 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:29.089389    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:29.426412 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:29.418993    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"]
Nov 03 13:36:29.426412 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:36:29.420503    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c3bfed_904b_4b38_ada6_3d3dd8c80c51.slice/crio-205c40fea0fbd8a9ac7f926807e1d7a3a09bdffaae8aaab9632aa11c8510102d WatchSource:0}: Error finding container 205c40fea0fbd8a9ac7f926807e1d7a3a09bdffaae8aaab9632aa11c8510102d: Status 404 returned error can't find the container with id 205c40fea0fbd8a9ac7f926807e1d7a3a09bdffaae8aaab9632aa11c8510102d
Nov 03 13:36:29.552453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:29.552417    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj" event={"ID":"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51","Type":"ContainerStarted","Data":"205c40fea0fbd8a9ac7f926807e1d7a3a09bdffaae8aaab9632aa11c8510102d"}
Nov 03 13:36:30.562415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.562322    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj" event={"ID":"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51","Type":"ContainerStarted","Data":"077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6"}
Nov 03 13:36:30.562825 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.562489    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:30.592904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.592841    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj" podStartSLOduration=2.186240141 podStartE2EDuration="2.592825881s" totalImagesPullingTime="406.58574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:30.590669293 +0000 UTC m=+1678.402337482" watchObservedRunningTime="2026-11-03 13:36:30.592825881 +0000 UTC m=+1678.404494069"
Nov 03 13:36:30.869455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.869358    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"]
Nov 03 13:36:30.877197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.877165    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:30.895905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.895877    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1ec4b892-openshift-default-dockercfg-rm5mf\""
Nov 03 13:36:30.901969 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.899965    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"]
Nov 03 13:36:30.931867 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.931826    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:30.931867 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.931878    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-data\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:30.932157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.932072    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-envoy\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:30.932157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.932104    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-istio-token\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:30.932157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.932124    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63038f07-11c9-4353-9444-fe92469f16c1-istiod-ca-cert\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:30.932318 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.932216    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63038f07-11c9-4353-9444-fe92469f16c1-istio-podinfo\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:30.932397 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.932314    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-certs\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:30.932397 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.932377    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-kube-api-access-xdknm\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:30.932536 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:30.932517    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-credential-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034266    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-certs\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034322    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-kube-api-access-xdknm\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034465    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-credential-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034634 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034519    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034639    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-data\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034790 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034768    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-certs\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034857 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034830    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034895    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-credential-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034902    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-envoy\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.034973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.034965    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-istio-token\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.035086 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.035000    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63038f07-11c9-4353-9444-fe92469f16c1-istiod-ca-cert\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.035086 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.035021    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-data\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.035086 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.035035    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63038f07-11c9-4353-9444-fe92469f16c1-istio-podinfo\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.035724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.035701    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63038f07-11c9-4353-9444-fe92469f16c1-istiod-ca-cert\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.037477 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.037451    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63038f07-11c9-4353-9444-fe92469f16c1-istio-podinfo\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.037606 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.037586    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-envoy\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.047748 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.047714    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-istio-token\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.048196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.048165    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-kube-api-access-xdknm\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.200915 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.200875    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:31.507437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.507409    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"]
Nov 03 13:36:31.509502 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:36:31.509470    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63038f07_11c9_4353_9444_fe92469f16c1.slice/crio-694e9c735938caff6b1dd73e785c059950728f3fcb8d31c91d21cd7ca9074ac1 WatchSource:0}: Error finding container 694e9c735938caff6b1dd73e785c059950728f3fcb8d31c91d21cd7ca9074ac1: Status 404 returned error can't find the container with id 694e9c735938caff6b1dd73e785c059950728f3fcb8d31c91d21cd7ca9074ac1
Nov 03 13:36:31.512526 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.512494    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:36:31.512601 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.512561    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:36:31.512601 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.512590    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:36:31.572405 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:31.572366    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" event={"ID":"63038f07-11c9-4353-9444-fe92469f16c1","Type":"ContainerStarted","Data":"694e9c735938caff6b1dd73e785c059950728f3fcb8d31c91d21cd7ca9074ac1"}
Nov 03 13:36:32.583366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:32.582838    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" event={"ID":"63038f07-11c9-4353-9444-fe92469f16c1","Type":"ContainerStarted","Data":"331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a"}
Nov 03 13:36:32.609870 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:32.609804    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" podStartSLOduration=2.609784598 podStartE2EDuration="2.609784598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:32.607318512 +0000 UTC m=+1680.418986720" watchObservedRunningTime="2026-11-03 13:36:32.609784598 +0000 UTC m=+1680.421452787"
Nov 03 13:36:33.204502 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:33.204465    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:33.206454 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:33.206426    4447 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 13:36:33.206577 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:33.206478    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" podUID="63038f07-11c9-4353-9444-fe92469f16c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 13:36:33.414896 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:36:33.414861    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:36:34.204673 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:34.204634    4447 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 13:36:34.205075 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:34.204687    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" podUID="63038f07-11c9-4353-9444-fe92469f16c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 13:36:35.204274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:35.204234    4447 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 13:36:35.204474 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:35.204304    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" podUID="63038f07-11c9-4353-9444-fe92469f16c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 13:36:36.204194 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:36.204160    4447 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 13:36:36.204797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:36.204766    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" podUID="63038f07-11c9-4353-9444-fe92469f16c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 13:36:36.579265 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:36.579192    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:36:37.168559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.168516    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"]
Nov 03 13:36:37.172829 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.172521    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj" podUID="86c79098-7c03-49b1-a6a8-cb33818ce7ee" containerName="maas-api" containerID="cri-o://76beb0b8fd8466570627ff0e891d5142d7649ff96a62ff1cc9bb094bb0080f1e" gracePeriod=30
Nov 03 13:36:37.204679 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.204276    4447 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 13:36:37.204679 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.204338    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" podUID="63038f07-11c9-4353-9444-fe92469f16c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 13:36:37.712362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.710711    4447 generic.go:356] "Generic (PLEG): container finished" podID="86c79098-7c03-49b1-a6a8-cb33818ce7ee" containerID="76beb0b8fd8466570627ff0e891d5142d7649ff96a62ff1cc9bb094bb0080f1e" exitCode=0
Nov 03 13:36:37.712362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.710780    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj" event={"ID":"86c79098-7c03-49b1-a6a8-cb33818ce7ee","Type":"ContainerDied","Data":"76beb0b8fd8466570627ff0e891d5142d7649ff96a62ff1cc9bb094bb0080f1e"}
Nov 03 13:36:37.767913 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.767885    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:36:37.919891 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.919808    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-maas-api-tls\") pod \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") "
Nov 03 13:36:37.920056 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.919919    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-metrics-certs\") pod \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") "
Nov 03 13:36:37.920056 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.920004    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c79098-7c03-49b1-a6a8-cb33818ce7ee-kube-api-access-7k8cv\" (UniqueName: \"kubernetes.io/projected/86c79098-7c03-49b1-a6a8-cb33818ce7ee-kube-api-access-7k8cv\") pod \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") "
Nov 03 13:36:37.920185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.920090    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c79098-7c03-49b1-a6a8-cb33818ce7ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c79098-7c03-49b1-a6a8-cb33818ce7ee-openshift-service-ca\") pod \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\" (UID: \"86c79098-7c03-49b1-a6a8-cb33818ce7ee\") "
Nov 03 13:36:37.920580 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.920547    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c79098-7c03-49b1-a6a8-cb33818ce7ee-openshift-service-ca" pod "86c79098-7c03-49b1-a6a8-cb33818ce7ee" (UID: "86c79098-7c03-49b1-a6a8-cb33818ce7ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:36:37.920736 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.920717    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c79098-7c03-49b1-a6a8-cb33818ce7ee-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:37.922400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.922375    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-maas-api-tls" pod "86c79098-7c03-49b1-a6a8-cb33818ce7ee" (UID: "86c79098-7c03-49b1-a6a8-cb33818ce7ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:36:37.922632 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.922616    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c79098-7c03-49b1-a6a8-cb33818ce7ee-kube-api-access-7k8cv" pod "86c79098-7c03-49b1-a6a8-cb33818ce7ee" (UID: "86c79098-7c03-49b1-a6a8-cb33818ce7ee"). InnerVolumeSpecName "kube-api-access-7k8cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:37.922728 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:37.922709    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-metrics-certs" pod "86c79098-7c03-49b1-a6a8-cb33818ce7ee" (UID: "86c79098-7c03-49b1-a6a8-cb33818ce7ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:36:38.021692 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.021657    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:38.021692 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.021688    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86c79098-7c03-49b1-a6a8-cb33818ce7ee-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:38.021692 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.021700    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k8cv\" (UniqueName: \"kubernetes.io/projected/86c79098-7c03-49b1-a6a8-cb33818ce7ee-kube-api-access-7k8cv\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:38.294549 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.294515    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:38.294976 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.294946    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:38.295841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.295824    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:38.718707 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.718677    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"
Nov 03 13:36:38.718707 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.718700    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj" event={"ID":"86c79098-7c03-49b1-a6a8-cb33818ce7ee","Type":"ContainerDied","Data":"2f5b1a5e1978bff925f7b29269515b0cfdb7dd14d67eeb7b4e4883ec35ca98cc"}
Nov 03 13:36:38.718954 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.718756    4447 scope.go:122] "RemoveContainer" containerID="76beb0b8fd8466570627ff0e891d5142d7649ff96a62ff1cc9bb094bb0080f1e"
Nov 03 13:36:38.745982 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.745945    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"]
Nov 03 13:36:38.749533 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:38.749503    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1ec4b892-5657dc4cdd-2gmlj"]
Nov 03 13:36:40.326425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:40.326389    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c79098-7c03-49b1-a6a8-cb33818ce7ee" path="/var/lib/kubelet/pods/86c79098-7c03-49b1-a6a8-cb33818ce7ee/volumes"
Nov 03 13:36:42.881047 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:42.881001    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"]
Nov 03 13:36:42.881512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:42.881254    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" podUID="63038f07-11c9-4353-9444-fe92469f16c1" containerName="istio-proxy" containerID="cri-o://331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a" gracePeriod=30
Nov 03 13:36:48.188133 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.188104    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:48.277335 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277297    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-data\") pod \"63038f07-11c9-4353-9444-fe92469f16c1\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") "
Nov 03 13:36:48.277335 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277368    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-kube-api-access-xdknm\") pod \"63038f07-11c9-4353-9444-fe92469f16c1\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") "
Nov 03 13:36:48.277588 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277432    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-istio-token\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-istio-token\") pod \"63038f07-11c9-4353-9444-fe92469f16c1\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") "
Nov 03 13:36:48.277588 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277512    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63038f07-11c9-4353-9444-fe92469f16c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63038f07-11c9-4353-9444-fe92469f16c1-istio-podinfo\") pod \"63038f07-11c9-4353-9444-fe92469f16c1\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") "
Nov 03 13:36:48.277666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277603    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-data" pod "63038f07-11c9-4353-9444-fe92469f16c1" (UID: "63038f07-11c9-4353-9444-fe92469f16c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.277666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277637    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63038f07-11c9-4353-9444-fe92469f16c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63038f07-11c9-4353-9444-fe92469f16c1-istiod-ca-cert\") pod \"63038f07-11c9-4353-9444-fe92469f16c1\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") "
Nov 03 13:36:48.277786 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277752    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-envoy\") pod \"63038f07-11c9-4353-9444-fe92469f16c1\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") "
Nov 03 13:36:48.277838 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277805    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-certs\") pod \"63038f07-11c9-4353-9444-fe92469f16c1\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") "
Nov 03 13:36:48.277889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277840    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-credential-socket\") pod \"63038f07-11c9-4353-9444-fe92469f16c1\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") "
Nov 03 13:36:48.277889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.277877    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-socket\") pod \"63038f07-11c9-4353-9444-fe92469f16c1\" (UID: \"63038f07-11c9-4353-9444-fe92469f16c1\") "
Nov 03 13:36:48.278045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.278014    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63038f07-11c9-4353-9444-fe92469f16c1-istiod-ca-cert" pod "63038f07-11c9-4353-9444-fe92469f16c1" (UID: "63038f07-11c9-4353-9444-fe92469f16c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:36:48.278315 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.278263    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-certs" pod "63038f07-11c9-4353-9444-fe92469f16c1" (UID: "63038f07-11c9-4353-9444-fe92469f16c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.278527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.278503    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-socket" pod "63038f07-11c9-4353-9444-fe92469f16c1" (UID: "63038f07-11c9-4353-9444-fe92469f16c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.278841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.278687    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.278841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.278719    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.278841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.278735    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.278841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.278750    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63038f07-11c9-4353-9444-fe92469f16c1-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.278841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.278695    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-credential-socket" pod "63038f07-11c9-4353-9444-fe92469f16c1" (UID: "63038f07-11c9-4353-9444-fe92469f16c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.280830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.280283    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63038f07-11c9-4353-9444-fe92469f16c1-istio-podinfo" pod "63038f07-11c9-4353-9444-fe92469f16c1" (UID: "63038f07-11c9-4353-9444-fe92469f16c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:36:48.280830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.280467    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-envoy" pod "63038f07-11c9-4353-9444-fe92469f16c1" (UID: "63038f07-11c9-4353-9444-fe92469f16c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.280830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.280649    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-kube-api-access-xdknm" pod "63038f07-11c9-4353-9444-fe92469f16c1" (UID: "63038f07-11c9-4353-9444-fe92469f16c1"). InnerVolumeSpecName "kube-api-access-xdknm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:48.281061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.280832    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-istio-token" pod "63038f07-11c9-4353-9444-fe92469f16c1" (UID: "63038f07-11c9-4353-9444-fe92469f16c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:48.380613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.380566    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.380613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.380593    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63038f07-11c9-4353-9444-fe92469f16c1-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.380613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.380602    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.380613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.380613    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63038f07-11c9-4353-9444-fe92469f16c1-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.380613 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.380623    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/63038f07-11c9-4353-9444-fe92469f16c1-kube-api-access-xdknm\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.792712 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.792679    4447 generic.go:356] "Generic (PLEG): container finished" podID="63038f07-11c9-4353-9444-fe92469f16c1" containerID="331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a" exitCode=0
Nov 03 13:36:48.792923 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.792725    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"
Nov 03 13:36:48.792923 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.792757    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" event={"ID":"63038f07-11c9-4353-9444-fe92469f16c1","Type":"ContainerDied","Data":"331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a"}
Nov 03 13:36:48.792923 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.792812    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd" event={"ID":"63038f07-11c9-4353-9444-fe92469f16c1","Type":"ContainerDied","Data":"694e9c735938caff6b1dd73e785c059950728f3fcb8d31c91d21cd7ca9074ac1"}
Nov 03 13:36:48.792923 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.792833    4447 scope.go:122] "RemoveContainer" containerID="331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a"
Nov 03 13:36:48.803685 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.803661    4447 scope.go:122] "RemoveContainer" containerID="331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a"
Nov 03 13:36:48.803956 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:36:48.803938    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a\": container with ID starting with 331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a not found: ID does not exist" containerID="331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a"
Nov 03 13:36:48.804017 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.803967    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a"} err="failed to get container status \"331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a\": rpc error: code = NotFound desc = could not find container \"331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a\": container with ID starting with 331c63492e8fc0153e462321e621ad2e41d129469957ee32b0fea8ad962be72a not found: ID does not exist"
Nov 03 13:36:48.813555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.813530    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"]
Nov 03 13:36:48.821306 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:48.821274    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-x24wd"]
Nov 03 13:36:50.321855 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:36:50.321822    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63038f07-11c9-4353-9444-fe92469f16c1" path="/var/lib/kubelet/pods/63038f07-11c9-4353-9444-fe92469f16c1/volumes"
Nov 03 13:37:03.417798 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:37:03.417761    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:37:21.012576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:21.012532    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:37:21.013415 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:21.013378    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:37:33.418783 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:37:33.418744    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:37:53.877009 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:53.876973    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"]
Nov 03 13:37:53.877581 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:53.877257    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj" podUID="a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" containerName="maas-api" containerID="cri-o://077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6" gracePeriod=30
Nov 03 13:37:54.340618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.340586    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:37:54.348248 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.348220    4447 generic.go:356] "Generic (PLEG): container finished" podID="a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" containerID="077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6" exitCode=0
Nov 03 13:37:54.348434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.348268    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"
Nov 03 13:37:54.348434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.348307    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj" event={"ID":"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51","Type":"ContainerDied","Data":"077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6"}
Nov 03 13:37:54.348434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.348377    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj" event={"ID":"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51","Type":"ContainerDied","Data":"205c40fea0fbd8a9ac7f926807e1d7a3a09bdffaae8aaab9632aa11c8510102d"}
Nov 03 13:37:54.348434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.348408    4447 scope.go:122] "RemoveContainer" containerID="077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6"
Nov 03 13:37:54.353843 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.353811    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-kube-api-access-snh5s\" (UniqueName: \"kubernetes.io/projected/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-kube-api-access-snh5s\") pod \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") "
Nov 03 13:37:54.353965 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.353937    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-metrics-certs\") pod \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") "
Nov 03 13:37:54.354032 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.353999    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-openshift-service-ca\") pod \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") "
Nov 03 13:37:54.354273 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.354252    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls\") pod \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\" (UID: \"a5c3bfed-904b-4b38-ada6-3d3dd8c80c51\") "
Nov 03 13:37:54.354552 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.354525    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-openshift-service-ca" pod "a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" (UID: "a5c3bfed-904b-4b38-ada6-3d3dd8c80c51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:37:54.355025 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.355007    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:54.356103 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.356081    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-kube-api-access-snh5s" pod "a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" (UID: "a5c3bfed-904b-4b38-ada6-3d3dd8c80c51"). InnerVolumeSpecName "kube-api-access-snh5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:37:54.356800 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.356770    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-metrics-certs" pod "a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" (UID: "a5c3bfed-904b-4b38-ada6-3d3dd8c80c51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:37:54.357077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.357040    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls" pod "a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" (UID: "a5c3bfed-904b-4b38-ada6-3d3dd8c80c51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:37:54.362560 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.362323    4447 scope.go:122] "RemoveContainer" containerID="077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6"
Nov 03 13:37:54.362698 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:37:54.362672    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6\": container with ID starting with 077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6 not found: ID does not exist" containerID="077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6"
Nov 03 13:37:54.362765 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.362712    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6"} err="failed to get container status \"077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6\": rpc error: code = NotFound desc = could not find container \"077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6\": container with ID starting with 077673df696241460e297783ed0985933bdbfb02dea355958cf31f6bd7de2bd6 not found: ID does not exist"
Nov 03 13:37:54.456460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.456401    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:54.456460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.456439    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snh5s\" (UniqueName: \"kubernetes.io/projected/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-kube-api-access-snh5s\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:54.456460 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.456456    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:54.672630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.672586    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"]
Nov 03 13:37:54.676101 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:54.676072    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-146318-6b684595f9-86lmj"]
Nov 03 13:37:56.320901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:37:56.320869    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" path="/var/lib/kubelet/pods/a5c3bfed-904b-4b38-ada6-3d3dd8c80c51/volumes"
Nov 03 13:38:03.416136 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:38:03.416067    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:38:29.637942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:29.637908    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"]
Nov 03 13:38:29.640605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:29.638155    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy" containerID="cri-o://5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292" gracePeriod=30
Nov 03 13:38:33.229856 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.229822    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"]
Nov 03 13:38:33.230481 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.230466    4447 cpu_manager.go:414] "containerMap: removing container" podUID="86c79098-7c03-49b1-a6a8-cb33818ce7ee" containerName="maas-api"
Nov 03 13:38:33.230481 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.230481    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c79098-7c03-49b1-a6a8-cb33818ce7ee" containerName="maas-api"
Nov 03 13:38:33.230579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.230497    4447 cpu_manager.go:414] "containerMap: removing container" podUID="63038f07-11c9-4353-9444-fe92469f16c1" containerName="istio-proxy"
Nov 03 13:38:33.230579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.230502    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63038f07-11c9-4353-9444-fe92469f16c1" containerName="istio-proxy"
Nov 03 13:38:33.230579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.230524    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" containerName="maas-api"
Nov 03 13:38:33.230579 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.230530    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" containerName="maas-api"
Nov 03 13:38:33.230708 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.230632    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c3bfed-904b-4b38-ada6-3d3dd8c80c51" containerName="maas-api"
Nov 03 13:38:33.230708 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.230648    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c79098-7c03-49b1-a6a8-cb33818ce7ee" containerName="maas-api"
Nov 03 13:38:33.230708 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.230655    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="63038f07-11c9-4353-9444-fe92469f16c1" containerName="istio-proxy"
Nov 03 13:38:33.234173 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.234154    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.242889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.242866    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a724a6-openshift-default-dockercfg-l5hxx\""
Nov 03 13:38:33.254896 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.254872    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"]
Nov 03 13:38:33.301379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.296410    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.301379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.296569    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.301379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.296619    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.301379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.296681    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.301379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.296722    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.301379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.296745    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4deac-8242-44bb-9b15-f7f57d2f7145-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.301379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.296857    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.301379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.296880    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5w7b\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-kube-api-access-z5w7b\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.301379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.296962    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.398428 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.398396    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.398594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.398446    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.398594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.398471    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4deac-8242-44bb-9b15-f7f57d2f7145-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.398701 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.398660    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.398767 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.398706    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5w7b\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-kube-api-access-z5w7b\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.398820 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.398780    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.398872 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.398859    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.399004 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.398989    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.399064 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.399052    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.399125 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.399077    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.399401 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.399375    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.399538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.399499    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.399651 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.399625    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.399954 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.399752    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4deac-8242-44bb-9b15-f7f57d2f7145-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.401659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.401637    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.401880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.401861    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.406825 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.406803    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.407399 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.407334    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5w7b\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-kube-api-access-z5w7b\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.417625 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:38:33.417564    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:38:33.549854 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.549773    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:33.850400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.850372    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"]
Nov 03 13:38:33.856674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.856640    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:38:33.856798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.856727    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:38:33.856798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:33.856775    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:38:34.040459 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.040421    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"]
Nov 03 13:38:34.045622 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.045598    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.074842 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.074780    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-2b8a08-openshift-default-dockercfg-dffsv\""
Nov 03 13:38:34.090793 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.080538    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"]
Nov 03 13:38:34.133234 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.133127    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.133441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.133242    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.133441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.133311    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.133441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.133367    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.133441 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.133436    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.133702 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.133481    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.133702 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.133538    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.133702 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.133589    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tvr\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-kube-api-access-28tvr\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.133868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.133713    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.245690 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.245461    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.245690 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.245534    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tvr\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-kube-api-access-28tvr\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.245690 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.245634    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.246374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.245755    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.246374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.245836    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.246374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.245892    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.246374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.245920    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.246374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.245974    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.246374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.246025    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.246875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.246771    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.247574 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.247045    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.253495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.248112    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.253495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.248622    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.253495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.249279    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.276005 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.275756    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.286101 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.286033    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.286698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.286656    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.288920 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.288893    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tvr\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-kube-api-access-28tvr\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.403331 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.403250    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:34.693354 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.693306    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" event={"ID":"aff4deac-8242-44bb-9b15-f7f57d2f7145","Type":"ContainerStarted","Data":"cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd"}
Nov 03 13:38:34.693516 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.693448    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" event={"ID":"aff4deac-8242-44bb-9b15-f7f57d2f7145","Type":"ContainerStarted","Data":"6a9322390f80bf12ab0bab336f0816fe828ec1b25225f9ffc0fe25247e157540"}
Nov 03 13:38:34.723154 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.718542    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" podStartSLOduration=1.718523109 podStartE2EDuration="1.718523109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:34.714742281 +0000 UTC m=+1802.526410498" watchObservedRunningTime="2026-11-03 13:38:34.718523109 +0000 UTC m=+1802.530191298"
Nov 03 13:38:34.727634 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:38:34.727446    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda62ab1b_1934_4d0b_84be_a9e1627b65e7.slice/crio-f2ee005532dc9693db71492fccd1e847f4b35a0e9994eec5e93f7580a28f9d67 WatchSource:0}: Error finding container f2ee005532dc9693db71492fccd1e847f4b35a0e9994eec5e93f7580a28f9d67: Status 404 returned error can't find the container with id f2ee005532dc9693db71492fccd1e847f4b35a0e9994eec5e93f7580a28f9d67
Nov 03 13:38:34.728804 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.728776    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"]
Nov 03 13:38:34.761365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.758868    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:38:34.761365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.759000    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:38:34.761365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:34.759067    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:38:35.171367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.165947    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:38:35.297860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.297805    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istiod-ca-cert\") pod \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") "
Nov 03 13:38:35.299797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.299501    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-data\") pod \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") "
Nov 03 13:38:35.299797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.299547    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-token\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-token\") pod \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") "
Nov 03 13:38:35.299797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.299666    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-kube-api-access-nkfmd\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-kube-api-access-nkfmd\") pod \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") "
Nov 03 13:38:35.299797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.299750    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-envoy\") pod \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") "
Nov 03 13:38:35.299797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.299791    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-socket\") pod \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") "
Nov 03 13:38:35.300068 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.299857    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-credential-socket\") pod \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") "
Nov 03 13:38:35.300068 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.299961    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-podinfo\") pod \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") "
Nov 03 13:38:35.300068 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.300009    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-certs\") pod \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\" (UID: \"a84bdc1b-5569-46f1-811a-2fe1ce4597b6\") "
Nov 03 13:38:35.302374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.298604    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istiod-ca-cert" pod "a84bdc1b-5569-46f1-811a-2fe1ce4597b6" (UID: "a84bdc1b-5569-46f1-811a-2fe1ce4597b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:35.302374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.301025    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-certs" pod "a84bdc1b-5569-46f1-811a-2fe1ce4597b6" (UID: "a84bdc1b-5569-46f1-811a-2fe1ce4597b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:35.302374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.301568    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-socket" pod "a84bdc1b-5569-46f1-811a-2fe1ce4597b6" (UID: "a84bdc1b-5569-46f1-811a-2fe1ce4597b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:35.302374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.301724    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-data" pod "a84bdc1b-5569-46f1-811a-2fe1ce4597b6" (UID: "a84bdc1b-5569-46f1-811a-2fe1ce4597b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:35.302374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.302274    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-credential-socket" pod "a84bdc1b-5569-46f1-811a-2fe1ce4597b6" (UID: "a84bdc1b-5569-46f1-811a-2fe1ce4597b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:35.311965 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.311933    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-envoy" pod "a84bdc1b-5569-46f1-811a-2fe1ce4597b6" (UID: "a84bdc1b-5569-46f1-811a-2fe1ce4597b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:35.312193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.312134    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-kube-api-access-nkfmd" pod "a84bdc1b-5569-46f1-811a-2fe1ce4597b6" (UID: "a84bdc1b-5569-46f1-811a-2fe1ce4597b6"). InnerVolumeSpecName "kube-api-access-nkfmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:35.312338 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.312322    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-podinfo" pod "a84bdc1b-5569-46f1-811a-2fe1ce4597b6" (UID: "a84bdc1b-5569-46f1-811a-2fe1ce4597b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:38:35.314987 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.314960    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-token" pod "a84bdc1b-5569-46f1-811a-2fe1ce4597b6" (UID: "a84bdc1b-5569-46f1-811a-2fe1ce4597b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:35.402272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.402184    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkfmd\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-kube-api-access-nkfmd\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.402272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.402221    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.402272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.402235    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.402272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.402247    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.402272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.402261    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.402272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.402273    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.402615 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.402286    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.402615 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.402298    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.402615 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.402310    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84bdc1b-5569-46f1-811a-2fe1ce4597b6-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.553624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.553541    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:35.556106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.556075    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 13:38:35.556316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.556288    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 13:38:35.717683 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.717644    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" event={"ID":"da62ab1b-1934-4d0b-84be-a9e1627b65e7","Type":"ContainerStarted","Data":"b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc"}
Nov 03 13:38:35.717957 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.717941    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" event={"ID":"da62ab1b-1934-4d0b-84be-a9e1627b65e7","Type":"ContainerStarted","Data":"f2ee005532dc9693db71492fccd1e847f4b35a0e9994eec5e93f7580a28f9d67"}
Nov 03 13:38:35.723322 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.723287    4447 generic.go:356] "Generic (PLEG): container finished" podID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerID="5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292" exitCode=0
Nov 03 13:38:35.724311 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.724292    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"
Nov 03 13:38:35.729288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.726443    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" event={"ID":"a84bdc1b-5569-46f1-811a-2fe1ce4597b6","Type":"ContainerDied","Data":"5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292"}
Nov 03 13:38:35.729288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.726497    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h" event={"ID":"a84bdc1b-5569-46f1-811a-2fe1ce4597b6","Type":"ContainerDied","Data":"56b61320baa810675ec047f4a80bbd18e6dbb4ba886d552997eb15a4250af45f"}
Nov 03 13:38:35.729288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.726532    4447 scope.go:122] "RemoveContainer" containerID="5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292"
Nov 03 13:38:35.747362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.742824    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" podStartSLOduration=1.7428066009999998 podStartE2EDuration="1.742806601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:35.741830407 +0000 UTC m=+1803.553498595" watchObservedRunningTime="2026-11-03 13:38:35.742806601 +0000 UTC m=+1803.554474790"
Nov 03 13:38:35.815247 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.815065    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"]
Nov 03 13:38:35.821007 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.820671    4447 scope.go:122] "RemoveContainer" containerID="5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292"
Nov 03 13:38:35.821377 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:38:35.821255    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292\": container with ID starting with 5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292 not found: ID does not exist" containerID="5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292"
Nov 03 13:38:35.821377 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.821294    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292"} err="failed to get container status \"5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292\": rpc error: code = NotFound desc = could not find container \"5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292\": container with ID starting with 5b9bea26826aa624a87e7c319e08bbb0d81c5631be72fd70f74600028f470292 not found: ID does not exist"
Nov 03 13:38:35.822448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:35.822378    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-qfc7h"]
Nov 03 13:38:36.324916 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:36.324884    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" path="/var/lib/kubelet/pods/a84bdc1b-5569-46f1-811a-2fe1ce4597b6/volumes"
Nov 03 13:38:36.403628 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:36.403591    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:36.405318 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:36.405290    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:38:36.405490 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:36.405336    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 13:38:36.557369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:36.553447    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 13:38:36.557369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:36.553510    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 13:38:37.416636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:37.416595    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:38:37.417142 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:37.416659    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 13:38:37.555491 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:37.555454    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 13:38:37.564837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:37.555750    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 13:38:38.186372 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:38.174053    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"]
Nov 03 13:38:38.188257 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:38.188224    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy" containerID="cri-o://8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9" gracePeriod=30
Nov 03 13:38:38.409828 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:38.409459    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:38:38.409828 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:38.409538    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 13:38:38.581430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:38.575909    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 13:38:38.581430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:38.575981    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 13:38:39.404353 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:39.404296    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:38:39.404779 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:39.404435    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 13:38:39.558379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:39.552760    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 13:38:39.558379 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:39.552826    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 13:38:40.403809 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:40.403767    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:38:40.404205 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:40.403838    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 13:38:40.551305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:40.550853    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 13:38:40.551305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:40.550919    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 13:38:41.404102 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:41.404065    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:38:41.404563 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:41.404134    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 13:38:41.723499 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:41.719702    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:41.724424 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:41.724094    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:41.727844 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:41.727815    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:38:42.408465 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:42.408434    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:42.408991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:42.408768    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:42.410018 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:42.409997    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:38:43.506052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.506026    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:38:43.631468 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631428    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-istio-token\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-istio-token\") pod \"785077ae-674b-43c8-b7f5-34ee6278b212\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") "
Nov 03 13:38:43.631659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631484    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/785077ae-674b-43c8-b7f5-34ee6278b212-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785077ae-674b-43c8-b7f5-34ee6278b212-istiod-ca-cert\") pod \"785077ae-674b-43c8-b7f5-34ee6278b212\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") "
Nov 03 13:38:43.631659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631528    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-certs\") pod \"785077ae-674b-43c8-b7f5-34ee6278b212\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") "
Nov 03 13:38:43.631659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631568    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/785077ae-674b-43c8-b7f5-34ee6278b212-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785077ae-674b-43c8-b7f5-34ee6278b212-istio-podinfo\") pod \"785077ae-674b-43c8-b7f5-34ee6278b212\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") "
Nov 03 13:38:43.631659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631634    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-envoy\") pod \"785077ae-674b-43c8-b7f5-34ee6278b212\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") "
Nov 03 13:38:43.631886 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631668    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-data\") pod \"785077ae-674b-43c8-b7f5-34ee6278b212\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") "
Nov 03 13:38:43.631886 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631852    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-kube-api-access-ndmcb\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-kube-api-access-ndmcb\") pod \"785077ae-674b-43c8-b7f5-34ee6278b212\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") "
Nov 03 13:38:43.631995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631879    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-certs" pod "785077ae-674b-43c8-b7f5-34ee6278b212" (UID: "785077ae-674b-43c8-b7f5-34ee6278b212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:43.631995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631912    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-credential-socket\") pod \"785077ae-674b-43c8-b7f5-34ee6278b212\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") "
Nov 03 13:38:43.631995 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631975    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-data" pod "785077ae-674b-43c8-b7f5-34ee6278b212" (UID: "785077ae-674b-43c8-b7f5-34ee6278b212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:43.632166 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631997    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-socket\") pod \"785077ae-674b-43c8-b7f5-34ee6278b212\" (UID: \"785077ae-674b-43c8-b7f5-34ee6278b212\") "
Nov 03 13:38:43.632315 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.631994    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785077ae-674b-43c8-b7f5-34ee6278b212-istiod-ca-cert" pod "785077ae-674b-43c8-b7f5-34ee6278b212" (UID: "785077ae-674b-43c8-b7f5-34ee6278b212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:43.632315 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.632180    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-credential-socket" pod "785077ae-674b-43c8-b7f5-34ee6278b212" (UID: "785077ae-674b-43c8-b7f5-34ee6278b212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:43.632315 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.632257    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-socket" pod "785077ae-674b-43c8-b7f5-34ee6278b212" (UID: "785077ae-674b-43c8-b7f5-34ee6278b212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:43.632707 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.632677    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:43.632707 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.632708    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785077ae-674b-43c8-b7f5-34ee6278b212-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:43.632924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.632723    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:43.632924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.632738    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:43.632924 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.632751    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:43.634077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.634053    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/785077ae-674b-43c8-b7f5-34ee6278b212-istio-podinfo" pod "785077ae-674b-43c8-b7f5-34ee6278b212" (UID: "785077ae-674b-43c8-b7f5-34ee6278b212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:38:43.634208 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.634092    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-kube-api-access-ndmcb" pod "785077ae-674b-43c8-b7f5-34ee6278b212" (UID: "785077ae-674b-43c8-b7f5-34ee6278b212"). InnerVolumeSpecName "kube-api-access-ndmcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:43.634208 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.634168    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-istio-token" pod "785077ae-674b-43c8-b7f5-34ee6278b212" (UID: "785077ae-674b-43c8-b7f5-34ee6278b212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:43.634558 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.634538    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-envoy" pod "785077ae-674b-43c8-b7f5-34ee6278b212" (UID: "785077ae-674b-43c8-b7f5-34ee6278b212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:43.734484 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.734399    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:43.734484 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.734441    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785077ae-674b-43c8-b7f5-34ee6278b212-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:43.734484 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.734457    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785077ae-674b-43c8-b7f5-34ee6278b212-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:43.734484 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.734472    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndmcb\" (UniqueName: \"kubernetes.io/projected/785077ae-674b-43c8-b7f5-34ee6278b212-kube-api-access-ndmcb\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:43.913264 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.913230    4447 generic.go:356] "Generic (PLEG): container finished" podID="785077ae-674b-43c8-b7f5-34ee6278b212" containerID="8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9" exitCode=0
Nov 03 13:38:43.913473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.913290    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"
Nov 03 13:38:43.913473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.913321    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" event={"ID":"785077ae-674b-43c8-b7f5-34ee6278b212","Type":"ContainerDied","Data":"8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9"}
Nov 03 13:38:43.913473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.913416    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l" event={"ID":"785077ae-674b-43c8-b7f5-34ee6278b212","Type":"ContainerDied","Data":"1e6992663dfa14c029405f4dc4160c9b21161632e75830ea99d830b5128100b5"}
Nov 03 13:38:43.913473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.913440    4447 scope.go:122] "RemoveContainer" containerID="8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9"
Nov 03 13:38:43.924845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.924826    4447 scope.go:122] "RemoveContainer" containerID="8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9"
Nov 03 13:38:43.925119 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:38:43.925098    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9\": container with ID starting with 8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9 not found: ID does not exist" containerID="8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9"
Nov 03 13:38:43.925183 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.925134    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9"} err="failed to get container status \"8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9\": rpc error: code = NotFound desc = could not find container \"8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9\": container with ID starting with 8bf7f8d0750050200337690c2de498751ce5f69130320c521d9cb2c9fca8f5a9 not found: ID does not exist"
Nov 03 13:38:43.951516 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.951480    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"]
Nov 03 13:38:43.957880 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:43.957847    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nhv7l"]
Nov 03 13:38:44.321727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:38:44.321693    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" path="/var/lib/kubelet/pods/785077ae-674b-43c8-b7f5-34ee6278b212/volumes"
Nov 03 13:39:03.411587 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:39:03.411552    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:39:14.474244 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:14.474209    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"]
Nov 03 13:39:14.480017 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:14.479993    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"]
Nov 03 13:39:29.496310 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.496262    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg" podUID="875a3afa-0140-40a0-956c-32821edd1f8b" containerName="main" containerID="cri-o://d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a" gracePeriod=45
Nov 03 13:39:29.502798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.502769    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" podUID="8a27ae67-e693-4daa-a521-ca104653b406" containerName="main" containerID="cri-o://14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210" gracePeriod=45
Nov 03 13:39:29.807106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.807071    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:39:29.811537 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.811515    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:39:29.848197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848151    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-model-cache\") pod \"875a3afa-0140-40a0-956c-32821edd1f8b\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") "
Nov 03 13:39:29.848452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848284    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-tmp-dir\") pod \"875a3afa-0140-40a0-956c-32821edd1f8b\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") "
Nov 03 13:39:29.848452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848324    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-home\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-home\") pod \"8a27ae67-e693-4daa-a521-ca104653b406\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") "
Nov 03 13:39:29.848452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848391    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-tmp-dir\") pod \"8a27ae67-e693-4daa-a521-ca104653b406\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") "
Nov 03 13:39:29.848452 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848432    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875a3afa-0140-40a0-956c-32821edd1f8b-kube-api-access-wbwrp\" (UniqueName: \"kubernetes.io/projected/875a3afa-0140-40a0-956c-32821edd1f8b-kube-api-access-wbwrp\") pod \"875a3afa-0140-40a0-956c-32821edd1f8b\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") "
Nov 03 13:39:29.848798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848480    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/875a3afa-0140-40a0-956c-32821edd1f8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/875a3afa-0140-40a0-956c-32821edd1f8b-tls-certs\") pod \"875a3afa-0140-40a0-956c-32821edd1f8b\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") "
Nov 03 13:39:29.848798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848519    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a27ae67-e693-4daa-a521-ca104653b406-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27ae67-e693-4daa-a521-ca104653b406-tls-certs\") pod \"8a27ae67-e693-4daa-a521-ca104653b406\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") "
Nov 03 13:39:29.848798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848626    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-model-cache\") pod \"8a27ae67-e693-4daa-a521-ca104653b406\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") "
Nov 03 13:39:29.848798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848656    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-dshm\") pod \"875a3afa-0140-40a0-956c-32821edd1f8b\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") "
Nov 03 13:39:29.848798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848765    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-dshm\") pod \"8a27ae67-e693-4daa-a521-ca104653b406\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") "
Nov 03 13:39:29.849185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848847    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a27ae67-e693-4daa-a521-ca104653b406-kube-api-access-8rqh7\" (UniqueName: \"kubernetes.io/projected/8a27ae67-e693-4daa-a521-ca104653b406-kube-api-access-8rqh7\") pod \"8a27ae67-e693-4daa-a521-ca104653b406\" (UID: \"8a27ae67-e693-4daa-a521-ca104653b406\") "
Nov 03 13:39:29.849185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.848886    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-home\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-home\") pod \"875a3afa-0140-40a0-956c-32821edd1f8b\" (UID: \"875a3afa-0140-40a0-956c-32821edd1f8b\") "
Nov 03 13:39:29.851432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.851361    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-tmp-dir" pod "875a3afa-0140-40a0-956c-32821edd1f8b" (UID: "875a3afa-0140-40a0-956c-32821edd1f8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.851556 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.851514    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-home" pod "8a27ae67-e693-4daa-a521-ca104653b406" (UID: "8a27ae67-e693-4daa-a521-ca104653b406"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.852487 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.852461    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-model-cache" pod "8a27ae67-e693-4daa-a521-ca104653b406" (UID: "8a27ae67-e693-4daa-a521-ca104653b406"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.852590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.852524    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-model-cache" pod "875a3afa-0140-40a0-956c-32821edd1f8b" (UID: "875a3afa-0140-40a0-956c-32821edd1f8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.853334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.852716    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-tmp-dir" pod "8a27ae67-e693-4daa-a521-ca104653b406" (UID: "8a27ae67-e693-4daa-a521-ca104653b406"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.861037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.857823    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-home" pod "875a3afa-0140-40a0-956c-32821edd1f8b" (UID: "875a3afa-0140-40a0-956c-32821edd1f8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.861037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.860444    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-dshm" pod "875a3afa-0140-40a0-956c-32821edd1f8b" (UID: "875a3afa-0140-40a0-956c-32821edd1f8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.863057 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.862027    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-dshm" pod "8a27ae67-e693-4daa-a521-ca104653b406" (UID: "8a27ae67-e693-4daa-a521-ca104653b406"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.863057 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.862047    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a27ae67-e693-4daa-a521-ca104653b406-kube-api-access-8rqh7" pod "8a27ae67-e693-4daa-a521-ca104653b406" (UID: "8a27ae67-e693-4daa-a521-ca104653b406"). InnerVolumeSpecName "kube-api-access-8rqh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:29.863057 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.862168    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875a3afa-0140-40a0-956c-32821edd1f8b-kube-api-access-wbwrp" pod "875a3afa-0140-40a0-956c-32821edd1f8b" (UID: "875a3afa-0140-40a0-956c-32821edd1f8b"). InnerVolumeSpecName "kube-api-access-wbwrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:29.863506 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.863176    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875a3afa-0140-40a0-956c-32821edd1f8b-tls-certs" pod "875a3afa-0140-40a0-956c-32821edd1f8b" (UID: "875a3afa-0140-40a0-956c-32821edd1f8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:39:29.863792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.863773    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a27ae67-e693-4daa-a521-ca104653b406-tls-certs" pod "8a27ae67-e693-4daa-a521-ca104653b406" (UID: "8a27ae67-e693-4daa-a521-ca104653b406"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:39:29.950901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.950864    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-tmp-dir\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.950901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.950899    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-home\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.950901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.950908    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-tmp-dir\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.951198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.950918    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbwrp\" (UniqueName: \"kubernetes.io/projected/875a3afa-0140-40a0-956c-32821edd1f8b-kube-api-access-wbwrp\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.951198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.950931    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875a3afa-0140-40a0-956c-32821edd1f8b-tls-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.951198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.950968    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27ae67-e693-4daa-a521-ca104653b406-tls-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.951198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.950979    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-model-cache\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.951198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.950988    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-dshm\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.951198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.950996    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27ae67-e693-4daa-a521-ca104653b406-dshm\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.951198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.951006    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rqh7\" (UniqueName: \"kubernetes.io/projected/8a27ae67-e693-4daa-a521-ca104653b406-kube-api-access-8rqh7\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.951198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.951019    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-home\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.951198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:29.951031    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875a3afa-0140-40a0-956c-32821edd1f8b-model-cache\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:30.335929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.335901    4447 generic.go:356] "Generic (PLEG): container finished" podID="875a3afa-0140-40a0-956c-32821edd1f8b" containerID="d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a" exitCode=0
Nov 03 13:39:30.336061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.335942    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"
Nov 03 13:39:30.336061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.335985    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg" event={"ID":"875a3afa-0140-40a0-956c-32821edd1f8b","Type":"ContainerDied","Data":"d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a"}
Nov 03 13:39:30.336061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.336025    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg" event={"ID":"875a3afa-0140-40a0-956c-32821edd1f8b","Type":"ContainerDied","Data":"e610a1038b417136045160b6d57b4f80d0fdf5895c4dfb9c15b6a7837a1c44cc"}
Nov 03 13:39:30.336061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.336042    4447 scope.go:122] "RemoveContainer" containerID="d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a"
Nov 03 13:39:30.338141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.338112    4447 generic.go:356] "Generic (PLEG): container finished" podID="8a27ae67-e693-4daa-a521-ca104653b406" containerID="14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210" exitCode=0
Nov 03 13:39:30.338331 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.338163    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"
Nov 03 13:39:30.338331 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.338236    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" event={"ID":"8a27ae67-e693-4daa-a521-ca104653b406","Type":"ContainerDied","Data":"14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210"}
Nov 03 13:39:30.338331 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.338258    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4" event={"ID":"8a27ae67-e693-4daa-a521-ca104653b406","Type":"ContainerDied","Data":"1476841beec76a827e717fe9ce733efb680dc39a2d43c4faa7136fa03621c174"}
Nov 03 13:39:30.347548 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.347524    4447 scope.go:122] "RemoveContainer" containerID="d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a"
Nov 03 13:39:30.347826 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:39:30.347804    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a\": container with ID starting with d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a not found: ID does not exist" containerID="d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a"
Nov 03 13:39:30.347875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.347837    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a"} err="failed to get container status \"d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a\": rpc error: code = NotFound desc = could not find container \"d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a\": container with ID starting with d94d57003c661dc808c71bd6438e85da37b7c60a3bdca1b6732c6f33a803e07a not found: ID does not exist"
Nov 03 13:39:30.347875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.347855    4447 scope.go:122] "RemoveContainer" containerID="14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210"
Nov 03 13:39:30.354237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.354211    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"]
Nov 03 13:39:30.358500 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.358477    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-2-w4-c381a7-kserve-7fc65d6d79-r72bg"]
Nov 03 13:39:30.359179 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.359166    4447 scope.go:122] "RemoveContainer" containerID="14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210"
Nov 03 13:39:30.359489 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:39:30.359461    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210\": container with ID starting with 14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210 not found: ID does not exist" containerID="14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210"
Nov 03 13:39:30.359489 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.359484    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210"} err="failed to get container status \"14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210\": rpc error: code = NotFound desc = could not find container \"14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210\": container with ID starting with 14f12dbb8a3bc0f3068af82250af1ccde0fb75a01a2218d8379461df55e86210 not found: ID does not exist"
Nov 03 13:39:30.375178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.375149    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"]
Nov 03 13:39:30.380438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:30.380414    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/distinct-w4-c381a7-kserve-69575dc48-xtbg4"]
Nov 03 13:39:32.320918 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:32.320887    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875a3afa-0140-40a0-956c-32821edd1f8b" path="/var/lib/kubelet/pods/875a3afa-0140-40a0-956c-32821edd1f8b/volumes"
Nov 03 13:39:32.321395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:32.321379    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a27ae67-e693-4daa-a521-ca104653b406" path="/var/lib/kubelet/pods/8a27ae67-e693-4daa-a521-ca104653b406/volumes"
Nov 03 13:39:33.414586 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:39:33.414551    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:39:33.452544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.452508    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"]
Nov 03 13:39:33.454440 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.454402    4447 cpu_manager.go:414] "containerMap: removing container" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy"
Nov 03 13:39:33.454440 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.454430    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy"
Nov 03 13:39:33.454900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.454458    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy"
Nov 03 13:39:33.454900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.454466    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy"
Nov 03 13:39:33.454900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.454506    4447 cpu_manager.go:414] "containerMap: removing container" podUID="875a3afa-0140-40a0-956c-32821edd1f8b" containerName="main"
Nov 03 13:39:33.454900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.454514    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875a3afa-0140-40a0-956c-32821edd1f8b" containerName="main"
Nov 03 13:39:33.454900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.454580    4447 cpu_manager.go:414] "containerMap: removing container" podUID="8a27ae67-e693-4daa-a521-ca104653b406" containerName="main"
Nov 03 13:39:33.454900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.454587    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a27ae67-e693-4daa-a521-ca104653b406" containerName="main"
Nov 03 13:39:33.455231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.454997    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84bdc1b-5569-46f1-811a-2fe1ce4597b6" containerName="istio-proxy"
Nov 03 13:39:33.455231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.455046    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="875a3afa-0140-40a0-956c-32821edd1f8b" containerName="main"
Nov 03 13:39:33.455231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.455063    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a27ae67-e693-4daa-a521-ca104653b406" containerName="main"
Nov 03 13:39:33.455231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.455118    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="785077ae-674b-43c8-b7f5-34ee6278b212" containerName="istio-proxy"
Nov 03 13:39:33.463545 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.463525    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.485498 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.485465    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"]
Nov 03 13:39:33.499461 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.497389    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.499461 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.497495    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.499461 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.497535    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.499461 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.497672    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.499461 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.497815    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.499461 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.497877    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73dcbee-73b7-4bd7-9254-09db37d787c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.499461 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.497918    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.499461 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.498023    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdtq\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-kube-api-access-4sdtq\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.499461 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.498103    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.599172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.599085    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.599172 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.599140    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73dcbee-73b7-4bd7-9254-09db37d787c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.599539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.599177    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.599539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.599229    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdtq\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-kube-api-access-4sdtq\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.599539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.599298    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.599928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.599904    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.600009 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.599983    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.600009 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.600000    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73dcbee-73b7-4bd7-9254-09db37d787c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.600163 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.600025    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.600163 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.600052    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.600266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.600173    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.600266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.600257    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.600725 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.600684    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.600830 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.600739    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.602545 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.602514    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.602641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.602515    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.608806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.608782    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.608943 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.608925    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdtq\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-kube-api-access-4sdtq\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:33.788228 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:33.788190    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:34.088418 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:39:34.088387    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73dcbee_73b7_4bd7_9254_09db37d787c9.slice/crio-734ec0972a32f0c2a412e9a6ba7cb9b7caff2afb66b375394b3c1715737e8423 WatchSource:0}: Error finding container 734ec0972a32f0c2a412e9a6ba7cb9b7caff2afb66b375394b3c1715737e8423: Status 404 returned error can't find the container with id 734ec0972a32f0c2a412e9a6ba7cb9b7caff2afb66b375394b3c1715737e8423
Nov 03 13:39:34.089102 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:34.089063    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"]
Nov 03 13:39:34.092221 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:34.092182    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:39:34.092333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:34.092269    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:39:34.092333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:34.092312    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:39:34.387627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:34.387523    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" event={"ID":"e73dcbee-73b7-4bd7-9254-09db37d787c9","Type":"ContainerStarted","Data":"01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416"}
Nov 03 13:39:34.387627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:34.387585    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" event={"ID":"e73dcbee-73b7-4bd7-9254-09db37d787c9","Type":"ContainerStarted","Data":"734ec0972a32f0c2a412e9a6ba7cb9b7caff2afb66b375394b3c1715737e8423"}
Nov 03 13:39:34.412188 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:34.412124    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" podStartSLOduration=1.412101502 podStartE2EDuration="1.412101502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:34.409164037 +0000 UTC m=+1862.220832289" watchObservedRunningTime="2026-11-03 13:39:34.412101502 +0000 UTC m=+1862.223769691"
Nov 03 13:39:34.788815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:34.788779    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:35.800498 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:35.800461    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 13:39:35.801119 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:35.801082    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 13:39:36.789251 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:36.789214    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 13:39:36.789438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:36.789282    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 13:39:37.801875 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:37.801833    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 13:39:37.802428 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:37.801898    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 13:39:38.802155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:38.797789    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 13:39:38.802155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:38.797853    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 13:39:39.789273 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:39.789228    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 13:39:39.789515 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:39.789300    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 13:39:40.837680 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:40.837646    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:40.838171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:40.838103    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:40.839149 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:40.839125    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:39:56.052734 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:56.052687    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"]
Nov 03 13:39:56.053254 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:39:56.052997    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy" containerID="cri-o://292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13" gracePeriod=30
Nov 03 13:40:01.333788 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.333722    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:40:01.404308 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404260    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-credential-socket\") pod \"8ad2c8ea-c4be-479f-a357-bfa000088cda\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") "
Nov 03 13:40:01.404529 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404429    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad2c8ea-c4be-479f-a357-bfa000088cda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad2c8ea-c4be-479f-a357-bfa000088cda-istiod-ca-cert\") pod \"8ad2c8ea-c4be-479f-a357-bfa000088cda\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") "
Nov 03 13:40:01.404529 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404505    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-socket\") pod \"8ad2c8ea-c4be-479f-a357-bfa000088cda\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") "
Nov 03 13:40:01.404645 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404550    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-token\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-token\") pod \"8ad2c8ea-c4be-479f-a357-bfa000088cda\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") "
Nov 03 13:40:01.404709 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404585    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-envoy\") pod \"8ad2c8ea-c4be-479f-a357-bfa000088cda\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") "
Nov 03 13:40:01.404796 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404778    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-certs\") pod \"8ad2c8ea-c4be-479f-a357-bfa000088cda\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") "
Nov 03 13:40:01.404898 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404880    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-podinfo\") pod \"8ad2c8ea-c4be-479f-a357-bfa000088cda\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") "
Nov 03 13:40:01.404964 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404918    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-data\") pod \"8ad2c8ea-c4be-479f-a357-bfa000088cda\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") "
Nov 03 13:40:01.404964 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404960    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-kube-api-access-5pnrp\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-kube-api-access-5pnrp\") pod \"8ad2c8ea-c4be-479f-a357-bfa000088cda\" (UID: \"8ad2c8ea-c4be-479f-a357-bfa000088cda\") "
Nov 03 13:40:01.405310 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.404905    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-credential-socket" pod "8ad2c8ea-c4be-479f-a357-bfa000088cda" (UID: "8ad2c8ea-c4be-479f-a357-bfa000088cda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.405413 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.405092    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-socket" pod "8ad2c8ea-c4be-479f-a357-bfa000088cda" (UID: "8ad2c8ea-c4be-479f-a357-bfa000088cda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.405472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.405248    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-data" pod "8ad2c8ea-c4be-479f-a357-bfa000088cda" (UID: "8ad2c8ea-c4be-479f-a357-bfa000088cda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.405472 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.405283    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad2c8ea-c4be-479f-a357-bfa000088cda-istiod-ca-cert" pod "8ad2c8ea-c4be-479f-a357-bfa000088cda" (UID: "8ad2c8ea-c4be-479f-a357-bfa000088cda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:01.405576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.405470    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-certs" pod "8ad2c8ea-c4be-479f-a357-bfa000088cda" (UID: "8ad2c8ea-c4be-479f-a357-bfa000088cda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.406622 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.406038    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.406622 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.406517    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.406622 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.406540    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.406622 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.406566    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.406622 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.406581    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad2c8ea-c4be-479f-a357-bfa000088cda-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.408305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.408209    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-envoy" pod "8ad2c8ea-c4be-479f-a357-bfa000088cda" (UID: "8ad2c8ea-c4be-479f-a357-bfa000088cda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.408305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.408241    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-podinfo" pod "8ad2c8ea-c4be-479f-a357-bfa000088cda" (UID: "8ad2c8ea-c4be-479f-a357-bfa000088cda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:40:01.408305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.408282    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-kube-api-access-5pnrp" pod "8ad2c8ea-c4be-479f-a357-bfa000088cda" (UID: "8ad2c8ea-c4be-479f-a357-bfa000088cda"). InnerVolumeSpecName "kube-api-access-5pnrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:01.409964 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.409937    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-token" pod "8ad2c8ea-c4be-479f-a357-bfa000088cda" (UID: "8ad2c8ea-c4be-479f-a357-bfa000088cda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:01.509892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.509837    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pnrp\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-kube-api-access-5pnrp\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.509892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.509879    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.509892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.509894    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.509892 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.509908    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad2c8ea-c4be-479f-a357-bfa000088cda-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.593212 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.593115    4447 generic.go:356] "Generic (PLEG): container finished" podID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerID="292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13" exitCode=0
Nov 03 13:40:01.593212 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.593156    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"
Nov 03 13:40:01.593212 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.593193    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" event={"ID":"8ad2c8ea-c4be-479f-a357-bfa000088cda","Type":"ContainerDied","Data":"292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13"}
Nov 03 13:40:01.594019 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.593253    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q" event={"ID":"8ad2c8ea-c4be-479f-a357-bfa000088cda","Type":"ContainerDied","Data":"dec790faf3537cea4576cd16759db86d868e1e1119d290ad2fa7997fcc69f701"}
Nov 03 13:40:01.594019 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.593275    4447 scope.go:122] "RemoveContainer" containerID="292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13"
Nov 03 13:40:01.605053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.605028    4447 scope.go:122] "RemoveContainer" containerID="292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13"
Nov 03 13:40:01.605381 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:01.605357    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13\": container with ID starting with 292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13 not found: ID does not exist" containerID="292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13"
Nov 03 13:40:01.605464 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.605389    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13"} err="failed to get container status \"292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13\": rpc error: code = NotFound desc = could not find container \"292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13\": container with ID starting with 292152d7423d815f8b638830c56276ffdcc087c2d80e5ea8aeb72025ca136b13 not found: ID does not exist"
Nov 03 13:40:01.616911 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.616889    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"]
Nov 03 13:40:01.622878 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:01.622855    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-mxk7q"]
Nov 03 13:40:02.320527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:02.320494    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" path="/var/lib/kubelet/pods/8ad2c8ea-c4be-479f-a357-bfa000088cda/volumes"
Nov 03 13:40:03.412308 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:03.412269    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:40:33.413891 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:33.413854    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:40:45.458684 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.458595    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"]
Nov 03 13:40:45.459225 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.458814    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv" podUID="cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" containerName="maas-api" containerID="cri-o://c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa" gracePeriod=30
Nov 03 13:40:45.757266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.757206    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:40:45.851370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.849367    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-kube-api-access-72l9p\" (UniqueName: \"kubernetes.io/projected/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-kube-api-access-72l9p\") pod \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") "
Nov 03 13:40:45.851370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.849731    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-metrics-certs\") pod \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") "
Nov 03 13:40:45.851370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.849832    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-maas-api-tls\") pod \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") "
Nov 03 13:40:45.851370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.850048    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-openshift-service-ca\") pod \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\" (UID: \"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e\") "
Nov 03 13:40:45.851370 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.851203    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-openshift-service-ca" pod "cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" (UID: "cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:45.875431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.856560    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-maas-api-tls" pod "cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" (UID: "cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:40:45.907386 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.903477    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-metrics-certs" pod "cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" (UID: "cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:40:45.907386 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.903629    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-kube-api-access-72l9p" pod "cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" (UID: "cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e"). InnerVolumeSpecName "kube-api-access-72l9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:45.946167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.942877    4447 generic.go:356] "Generic (PLEG): container finished" podID="cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" containerID="c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa" exitCode=0
Nov 03 13:40:45.946167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.943104    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv" event={"ID":"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e","Type":"ContainerDied","Data":"c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa"}
Nov 03 13:40:45.946167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.943320    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"
Nov 03 13:40:45.949407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.947454    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv" event={"ID":"cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e","Type":"ContainerDied","Data":"762c579e61b267d184f0292ccf4fd622ca18cf946a4d052938cd96c81706212b"}
Nov 03 13:40:45.949407 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.947513    4447 scope.go:122] "RemoveContainer" containerID="c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa"
Nov 03 13:40:45.954984 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.954887    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72l9p\" (UniqueName: \"kubernetes.io/projected/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-kube-api-access-72l9p\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:45.954984 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.954920    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:45.954984 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.954936    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:45.954984 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.954950    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:45.977374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.977330    4447 scope.go:122] "RemoveContainer" containerID="c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa"
Nov 03 13:40:45.987584 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:45.979481    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa\": container with ID starting with c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa not found: ID does not exist" containerID="c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa"
Nov 03 13:40:45.987584 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.987558    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa"} err="failed to get container status \"c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa\": rpc error: code = NotFound desc = could not find container \"c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa\": container with ID starting with c4335d890ea6a00605208cf22f385ac16e211aa813f615d14be3534e55ca55aa not found: ID does not exist"
Nov 03 13:40:45.991139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.991064    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"]
Nov 03 13:40:45.998744 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:45.998706    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a62d22-5d8d5fccdc-9lvhv"]
Nov 03 13:40:46.354418 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:46.352912    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" path="/var/lib/kubelet/pods/cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e/volumes"
Nov 03 13:40:50.562687 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.562650    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-d9597e"]
Nov 03 13:40:50.563422 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.563404    4447 cpu_manager.go:414] "containerMap: removing container" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy"
Nov 03 13:40:50.563422 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.563422    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy"
Nov 03 13:40:50.563512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.563444    4447 cpu_manager.go:414] "containerMap: removing container" podUID="cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" containerName="maas-api"
Nov 03 13:40:50.563512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.563450    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" containerName="maas-api"
Nov 03 13:40:50.563599 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.563540    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9bbb1b-dadb-4d79-a999-dc4f32fcb64e" containerName="maas-api"
Nov 03 13:40:50.563599 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.563555    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad2c8ea-c4be-479f-a357-bfa000088cda" containerName="istio-proxy"
Nov 03 13:40:50.568722 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.568697    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d9597e"
Nov 03 13:40:50.574026 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.573733    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-d9597e"]
Nov 03 13:40:50.638672 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.635099    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncr7v\""
Nov 03 13:40:50.669327 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.669293    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcnq\" (UniqueName: \"kubernetes.io/projected/0a346bfa-8935-4d53-b150-49c5ccc23266-kube-api-access-sdcnq\") pod \"test-curl-7095-d9597e\" (UID: \"0a346bfa-8935-4d53-b150-49c5ccc23266\") " pod="opendatahub/test-curl-7095-d9597e"
Nov 03 13:40:50.771144 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.771106    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcnq\" (UniqueName: \"kubernetes.io/projected/0a346bfa-8935-4d53-b150-49c5ccc23266-kube-api-access-sdcnq\") pod \"test-curl-7095-d9597e\" (UID: \"0a346bfa-8935-4d53-b150-49c5ccc23266\") " pod="opendatahub/test-curl-7095-d9597e"
Nov 03 13:40:50.779811 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.779781    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcnq\" (UniqueName: \"kubernetes.io/projected/0a346bfa-8935-4d53-b150-49c5ccc23266-kube-api-access-sdcnq\") pod \"test-curl-7095-d9597e\" (UID: \"0a346bfa-8935-4d53-b150-49c5ccc23266\") " pod="opendatahub/test-curl-7095-d9597e"
Nov 03 13:40:50.905397 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:50.894169    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d9597e"
Nov 03 13:40:51.217182 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:51.217153    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-d9597e"]
Nov 03 13:40:51.218303 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:40:51.218276    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a346bfa_8935_4d53_b150_49c5ccc23266.slice/crio-ba18d7588c358d6d2e99f759c2f293b20649a74f0612a60a8862011f4e300745 WatchSource:0}: Error finding container ba18d7588c358d6d2e99f759c2f293b20649a74f0612a60a8862011f4e300745: Status 404 returned error can't find the container with id ba18d7588c358d6d2e99f759c2f293b20649a74f0612a60a8862011f4e300745
Nov 03 13:40:52.027114 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.027076    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-d9597e" event={"ID":"0a346bfa-8935-4d53-b150-49c5ccc23266","Type":"ContainerStarted","Data":"278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d"}
Nov 03 13:40:52.027114 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.027129    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-d9597e" event={"ID":"0a346bfa-8935-4d53-b150-49c5ccc23266","Type":"ContainerStarted","Data":"ba18d7588c358d6d2e99f759c2f293b20649a74f0612a60a8862011f4e300745"}
Nov 03 13:40:52.042848 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.042798    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7095-d9597e" podStartSLOduration=2.042783221 podStartE2EDuration="2.042783221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:52.041506272 +0000 UTC m=+1939.853174486" watchObservedRunningTime="2026-11-03 13:40:52.042783221 +0000 UTC m=+1939.854451409"
Nov 03 13:40:52.758142 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.758105    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-d9597e"]
Nov 03 13:40:52.766391 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.766358    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-d9597e"]
Nov 03 13:40:52.920988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.920953    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-4b5732"]
Nov 03 13:40:52.921618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.921603    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0a346bfa-8935-4d53-b150-49c5ccc23266" containerName="test-curl-7095-d9597e"
Nov 03 13:40:52.921674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.921619    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a346bfa-8935-4d53-b150-49c5ccc23266" containerName="test-curl-7095-d9597e"
Nov 03 13:40:52.921736 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.921727    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a346bfa-8935-4d53-b150-49c5ccc23266" containerName="test-curl-7095-d9597e"
Nov 03 13:40:52.925287 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.925269    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-4b5732"
Nov 03 13:40:52.932497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:52.932473    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-4b5732"]
Nov 03 13:40:53.013988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:53.013901    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5g4j\" (UniqueName: \"kubernetes.io/projected/2b8a467f-21c4-4ea2-aa6a-371698773cea-kube-api-access-c5g4j\") pod \"test-curl-7095-4b5732\" (UID: \"2b8a467f-21c4-4ea2-aa6a-371698773cea\") " pod="opendatahub/test-curl-7095-4b5732"
Nov 03 13:40:53.115476 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:53.115438    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5g4j\" (UniqueName: \"kubernetes.io/projected/2b8a467f-21c4-4ea2-aa6a-371698773cea-kube-api-access-c5g4j\") pod \"test-curl-7095-4b5732\" (UID: \"2b8a467f-21c4-4ea2-aa6a-371698773cea\") " pod="opendatahub/test-curl-7095-4b5732"
Nov 03 13:40:53.123274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:53.123246    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5g4j\" (UniqueName: \"kubernetes.io/projected/2b8a467f-21c4-4ea2-aa6a-371698773cea-kube-api-access-c5g4j\") pod \"test-curl-7095-4b5732\" (UID: \"2b8a467f-21c4-4ea2-aa6a-371698773cea\") " pod="opendatahub/test-curl-7095-4b5732"
Nov 03 13:40:53.245984 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:53.245950    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-4b5732"
Nov 03 13:40:53.553934 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:53.553753    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-4b5732"]
Nov 03 13:40:53.582164 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:40:53.582127    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b8a467f_21c4_4ea2_aa6a_371698773cea.slice/crio-65e031c6a6c0757284e7366e8c9831f814b7dda8e30fd9dfcb8ad4f9f1b1f573 WatchSource:0}: Error finding container 65e031c6a6c0757284e7366e8c9831f814b7dda8e30fd9dfcb8ad4f9f1b1f573: Status 404 returned error can't find the container with id 65e031c6a6c0757284e7366e8c9831f814b7dda8e30fd9dfcb8ad4f9f1b1f573
Nov 03 13:40:54.041231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:54.041196    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-4b5732" event={"ID":"2b8a467f-21c4-4ea2-aa6a-371698773cea","Type":"ContainerStarted","Data":"de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d"}
Nov 03 13:40:54.041231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:54.041238    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-4b5732" event={"ID":"2b8a467f-21c4-4ea2-aa6a-371698773cea","Type":"ContainerStarted","Data":"65e031c6a6c0757284e7366e8c9831f814b7dda8e30fd9dfcb8ad4f9f1b1f573"}
Nov 03 13:40:54.041508 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:54.041244    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7095-d9597e" podUID="0a346bfa-8935-4d53-b150-49c5ccc23266" containerName="test-curl-7095-d9597e" containerID="cri-o://278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d" gracePeriod=2
Nov 03 13:40:54.043477 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:54.043453    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-d9597e\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="0a346bfa-8935-4d53-b150-49c5ccc23266" pod="opendatahub/test-curl-7095-d9597e"
Nov 03 13:40:54.060011 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:54.059960    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7095-4b5732" podStartSLOduration=2.059949277 podStartE2EDuration="2.059949277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:54.058473196 +0000 UTC m=+1941.870141404" watchObservedRunningTime="2026-11-03 13:40:54.059949277 +0000 UTC m=+1941.871617466"
Nov 03 13:40:54.060011 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:54.059995    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-d9597e\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="0a346bfa-8935-4d53-b150-49c5ccc23266" pod="opendatahub/test-curl-7095-d9597e"
Nov 03 13:40:54.746004 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:54.745971    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-4b5732"]
Nov 03 13:40:54.757650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:54.757615    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-4b5732"]
Nov 03 13:40:56.058485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:56.058446    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7095-4b5732" podUID="2b8a467f-21c4-4ea2-aa6a-371698773cea" containerName="test-curl-7095-4b5732" containerID="cri-o://de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d" gracePeriod=2
Nov 03 13:40:56.060117 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:56.060077    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-4b5732\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="2b8a467f-21c4-4ea2-aa6a-371698773cea" pod="opendatahub/test-curl-7095-4b5732"
Nov 03 13:40:56.241418 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:56.241391    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d9597e"
Nov 03 13:40:56.243077 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:56.243050    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7095-4b5732\" is forbidden: User \"system:node:ip-10-0-132-167.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-167.us-east-2.compute.internal' and this object" podUID="2b8a467f-21c4-4ea2-aa6a-371698773cea" pod="opendatahub/test-curl-7095-4b5732"
Nov 03 13:40:56.358124 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:56.358050    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a346bfa-8935-4d53-b150-49c5ccc23266-kube-api-access-sdcnq\" (UniqueName: \"kubernetes.io/projected/0a346bfa-8935-4d53-b150-49c5ccc23266-kube-api-access-sdcnq\") pod \"0a346bfa-8935-4d53-b150-49c5ccc23266\" (UID: \"0a346bfa-8935-4d53-b150-49c5ccc23266\") "
Nov 03 13:40:56.360310 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:56.360286    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a346bfa-8935-4d53-b150-49c5ccc23266-kube-api-access-sdcnq" pod "0a346bfa-8935-4d53-b150-49c5ccc23266" (UID: "0a346bfa-8935-4d53-b150-49c5ccc23266"). InnerVolumeSpecName "kube-api-access-sdcnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:56.459744 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:56.459710    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdcnq\" (UniqueName: \"kubernetes.io/projected/0a346bfa-8935-4d53-b150-49c5ccc23266-kube-api-access-sdcnq\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:57.066948 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:57.066909    4447 generic.go:356] "Generic (PLEG): container finished" podID="0a346bfa-8935-4d53-b150-49c5ccc23266" containerID="278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d" exitCode=137
Nov 03 13:40:57.067404 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:57.066958    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d9597e"
Nov 03 13:40:57.067404 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:57.066977    4447 scope.go:122] "RemoveContainer" containerID="278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d"
Nov 03 13:40:57.077902 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:57.077883    4447 scope.go:122] "RemoveContainer" containerID="278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d"
Nov 03 13:40:57.078169 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:57.078150    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d\": container with ID starting with 278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d not found: ID does not exist" containerID="278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d"
Nov 03 13:40:57.078222 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:57.078177    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d"} err="failed to get container status \"278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d\": rpc error: code = NotFound desc = could not find container \"278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d\": container with ID starting with 278764bfc5a1c3b139a194c71a7d412787915a28ec72a1389c84f0be9fc60d8d not found: ID does not exist"
Nov 03 13:40:58.290315 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:58.290289    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-4b5732"
Nov 03 13:40:58.320599 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:58.320523    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a346bfa-8935-4d53-b150-49c5ccc23266" path="/var/lib/kubelet/pods/0a346bfa-8935-4d53-b150-49c5ccc23266/volumes"
Nov 03 13:40:58.395324 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:58.388949    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8a467f-21c4-4ea2-aa6a-371698773cea-kube-api-access-c5g4j\" (UniqueName: \"kubernetes.io/projected/2b8a467f-21c4-4ea2-aa6a-371698773cea-kube-api-access-c5g4j\") pod \"2b8a467f-21c4-4ea2-aa6a-371698773cea\" (UID: \"2b8a467f-21c4-4ea2-aa6a-371698773cea\") "
Nov 03 13:40:58.414367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:58.401958    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8a467f-21c4-4ea2-aa6a-371698773cea-kube-api-access-c5g4j" pod "2b8a467f-21c4-4ea2-aa6a-371698773cea" (UID: "2b8a467f-21c4-4ea2-aa6a-371698773cea"). InnerVolumeSpecName "kube-api-access-c5g4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:58.491369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:58.491323    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5g4j\" (UniqueName: \"kubernetes.io/projected/2b8a467f-21c4-4ea2-aa6a-371698773cea-kube-api-access-c5g4j\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:59.084641 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:59.084608    4447 generic.go:356] "Generic (PLEG): container finished" podID="2b8a467f-21c4-4ea2-aa6a-371698773cea" containerID="de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d" exitCode=137
Nov 03 13:40:59.084868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:59.084666    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-4b5732"
Nov 03 13:40:59.084868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:59.084709    4447 scope.go:122] "RemoveContainer" containerID="de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d"
Nov 03 13:40:59.095818 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:59.095650    4447 scope.go:122] "RemoveContainer" containerID="de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d"
Nov 03 13:40:59.095962 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:40:59.095943    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d\": container with ID starting with de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d not found: ID does not exist" containerID="de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d"
Nov 03 13:40:59.096000 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:59.095973    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d"} err="failed to get container status \"de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d\": rpc error: code = NotFound desc = could not find container \"de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d\": container with ID starting with de734f42443527ae589bc525d21dda8d77c7fa0170d70d349f93b46e663f094d not found: ID does not exist"
Nov 03 13:40:59.776385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:59.776324    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"]
Nov 03 13:40:59.776762 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:40:59.776601    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766" podUID="73bc340b-451f-4c34-96ad-b514783d1f88" containerName="maas-api" containerID="cri-o://9e7ec687b647637969ae5f34088514188ebea8a9ca12be39c2efbbbe1aae765c" gracePeriod=30
Nov 03 13:41:00.159953 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.159809    4447 generic.go:356] "Generic (PLEG): container finished" podID="73bc340b-451f-4c34-96ad-b514783d1f88" containerID="9e7ec687b647637969ae5f34088514188ebea8a9ca12be39c2efbbbe1aae765c" exitCode=0
Nov 03 13:41:00.160357 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.160319    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766" event={"ID":"73bc340b-451f-4c34-96ad-b514783d1f88","Type":"ContainerDied","Data":"9e7ec687b647637969ae5f34088514188ebea8a9ca12be39c2efbbbe1aae765c"}
Nov 03 13:41:00.219708 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.219678    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:41:00.320432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.320395    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8a467f-21c4-4ea2-aa6a-371698773cea" path="/var/lib/kubelet/pods/2b8a467f-21c4-4ea2-aa6a-371698773cea/volumes"
Nov 03 13:41:00.351286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.351252    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bc340b-451f-4c34-96ad-b514783d1f88-kube-api-access-dbkbz\" (UniqueName: \"kubernetes.io/projected/73bc340b-451f-4c34-96ad-b514783d1f88-kube-api-access-dbkbz\") pod \"73bc340b-451f-4c34-96ad-b514783d1f88\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") "
Nov 03 13:41:00.351698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.351365    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bc340b-451f-4c34-96ad-b514783d1f88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bc340b-451f-4c34-96ad-b514783d1f88-openshift-service-ca\") pod \"73bc340b-451f-4c34-96ad-b514783d1f88\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") "
Nov 03 13:41:00.351856 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.351832    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bc340b-451f-4c34-96ad-b514783d1f88-openshift-service-ca" pod "73bc340b-451f-4c34-96ad-b514783d1f88" (UID: "73bc340b-451f-4c34-96ad-b514783d1f88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:00.352207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.352171    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-metrics-certs\") pod \"73bc340b-451f-4c34-96ad-b514783d1f88\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") "
Nov 03 13:41:00.352326 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.352248    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-maas-api-tls\") pod \"73bc340b-451f-4c34-96ad-b514783d1f88\" (UID: \"73bc340b-451f-4c34-96ad-b514783d1f88\") "
Nov 03 13:41:00.353316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.353293    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bc340b-451f-4c34-96ad-b514783d1f88-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:00.357845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.357789    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-maas-api-tls" pod "73bc340b-451f-4c34-96ad-b514783d1f88" (UID: "73bc340b-451f-4c34-96ad-b514783d1f88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:00.360825 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.360631    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bc340b-451f-4c34-96ad-b514783d1f88-kube-api-access-dbkbz" pod "73bc340b-451f-4c34-96ad-b514783d1f88" (UID: "73bc340b-451f-4c34-96ad-b514783d1f88"). InnerVolumeSpecName "kube-api-access-dbkbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:00.361023 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.361001    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-metrics-certs" pod "73bc340b-451f-4c34-96ad-b514783d1f88" (UID: "73bc340b-451f-4c34-96ad-b514783d1f88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:00.455175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.455130    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkbz\" (UniqueName: \"kubernetes.io/projected/73bc340b-451f-4c34-96ad-b514783d1f88-kube-api-access-dbkbz\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:00.455175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.455163    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:00.455175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:00.455178    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73bc340b-451f-4c34-96ad-b514783d1f88-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:01.178269 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:01.178223    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766" event={"ID":"73bc340b-451f-4c34-96ad-b514783d1f88","Type":"ContainerDied","Data":"a1ac0560c1fc19d1ee6c29ee53a30b1e9ab2f5811fe4b965f9b949dae04324f9"}
Nov 03 13:41:01.178841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:01.178372    4447 scope.go:122] "RemoveContainer" containerID="9e7ec687b647637969ae5f34088514188ebea8a9ca12be39c2efbbbe1aae765c"
Nov 03 13:41:01.178841 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:01.178482    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"
Nov 03 13:41:01.203632 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:01.203605    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"]
Nov 03 13:41:01.207207 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:01.207181    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cf53a4-5b49cff8cc-9g766"]
Nov 03 13:41:02.320636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:02.320604    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bc340b-451f-4c34-96ad-b514783d1f88" path="/var/lib/kubelet/pods/73bc340b-451f-4c34-96ad-b514783d1f88/volumes"
Nov 03 13:41:03.415866 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:41:03.415830    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:41:33.414020 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:41:33.413984    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:41:35.010052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.009526    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"]
Nov 03 13:41:35.010052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.009789    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt" podUID="4e665799-44f7-4a3d-9d6f-d9166accd2c9" containerName="maas-api" containerID="cri-o://c609794f0083036ec9928259fd4641f1b6fe4b325af4ecf996fbd8eb41300a16" gracePeriod=30
Nov 03 13:41:35.455582 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.455553    4447 generic.go:356] "Generic (PLEG): container finished" podID="4e665799-44f7-4a3d-9d6f-d9166accd2c9" containerID="c609794f0083036ec9928259fd4641f1b6fe4b325af4ecf996fbd8eb41300a16" exitCode=0
Nov 03 13:41:35.455705 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.455638    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt" event={"ID":"4e665799-44f7-4a3d-9d6f-d9166accd2c9","Type":"ContainerDied","Data":"c609794f0083036ec9928259fd4641f1b6fe4b325af4ecf996fbd8eb41300a16"}
Nov 03 13:41:35.455750 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.455704    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt" event={"ID":"4e665799-44f7-4a3d-9d6f-d9166accd2c9","Type":"ContainerDied","Data":"3bceee7070416057acdc9017e7b7d6f0365434cb7dccaa8144321325fdbaa9a6"}
Nov 03 13:41:35.455750 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.455732    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bceee7070416057acdc9017e7b7d6f0365434cb7dccaa8144321325fdbaa9a6"
Nov 03 13:41:35.476074 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.476042    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:41:35.538926 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.538888    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-maas-api-tls\") pod \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") "
Nov 03 13:41:35.539135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.538970    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-metrics-certs\") pod \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") "
Nov 03 13:41:35.539135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.539027    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e665799-44f7-4a3d-9d6f-d9166accd2c9-kube-api-access-x8rb7\" (UniqueName: \"kubernetes.io/projected/4e665799-44f7-4a3d-9d6f-d9166accd2c9-kube-api-access-x8rb7\") pod \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") "
Nov 03 13:41:35.539135 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.539088    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e665799-44f7-4a3d-9d6f-d9166accd2c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e665799-44f7-4a3d-9d6f-d9166accd2c9-openshift-service-ca\") pod \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\" (UID: \"4e665799-44f7-4a3d-9d6f-d9166accd2c9\") "
Nov 03 13:41:35.539627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.539595    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e665799-44f7-4a3d-9d6f-d9166accd2c9-openshift-service-ca" pod "4e665799-44f7-4a3d-9d6f-d9166accd2c9" (UID: "4e665799-44f7-4a3d-9d6f-d9166accd2c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:35.539812 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.539784    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e665799-44f7-4a3d-9d6f-d9166accd2c9-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:35.541330 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.541304    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-metrics-certs" pod "4e665799-44f7-4a3d-9d6f-d9166accd2c9" (UID: "4e665799-44f7-4a3d-9d6f-d9166accd2c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:35.541730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.541704    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-maas-api-tls" pod "4e665799-44f7-4a3d-9d6f-d9166accd2c9" (UID: "4e665799-44f7-4a3d-9d6f-d9166accd2c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:35.541817 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.541775    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e665799-44f7-4a3d-9d6f-d9166accd2c9-kube-api-access-x8rb7" pod "4e665799-44f7-4a3d-9d6f-d9166accd2c9" (UID: "4e665799-44f7-4a3d-9d6f-d9166accd2c9"). InnerVolumeSpecName "kube-api-access-x8rb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:35.641430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.641327    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:35.641430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.641395    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e665799-44f7-4a3d-9d6f-d9166accd2c9-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:35.641430 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:35.641406    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8rb7\" (UniqueName: \"kubernetes.io/projected/4e665799-44f7-4a3d-9d6f-d9166accd2c9-kube-api-access-x8rb7\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:36.462705 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:36.462673    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"
Nov 03 13:41:36.479514 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:36.479487    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"]
Nov 03 13:41:36.485009 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:36.484980    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d3ca48-748b5fdf8f-cg9dt"]
Nov 03 13:41:38.323845 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.323746    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e665799-44f7-4a3d-9d6f-d9166accd2c9" path="/var/lib/kubelet/pods/4e665799-44f7-4a3d-9d6f-d9166accd2c9/volumes"
Nov 03 13:41:38.440733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.440701    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"]
Nov 03 13:41:38.441580 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.441553    4447 cpu_manager.go:414] "containerMap: removing container" podUID="73bc340b-451f-4c34-96ad-b514783d1f88" containerName="maas-api"
Nov 03 13:41:38.441580 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.441579    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bc340b-451f-4c34-96ad-b514783d1f88" containerName="maas-api"
Nov 03 13:41:38.441768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.441633    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4e665799-44f7-4a3d-9d6f-d9166accd2c9" containerName="maas-api"
Nov 03 13:41:38.441768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.441639    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e665799-44f7-4a3d-9d6f-d9166accd2c9" containerName="maas-api"
Nov 03 13:41:38.441768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.441653    4447 cpu_manager.go:414] "containerMap: removing container" podUID="2b8a467f-21c4-4ea2-aa6a-371698773cea" containerName="test-curl-7095-4b5732"
Nov 03 13:41:38.441768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.441658    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8a467f-21c4-4ea2-aa6a-371698773cea" containerName="test-curl-7095-4b5732"
Nov 03 13:41:38.441768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.441768    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e665799-44f7-4a3d-9d6f-d9166accd2c9" containerName="maas-api"
Nov 03 13:41:38.441977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.441782    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8a467f-21c4-4ea2-aa6a-371698773cea" containerName="test-curl-7095-4b5732"
Nov 03 13:41:38.441977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.441794    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bc340b-451f-4c34-96ad-b514783d1f88" containerName="maas-api"
Nov 03 13:41:38.446155 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.446136    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.464986 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.464960    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"]
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.483579    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.483623    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.483655    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c95\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-kube-api-access-r5c95\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.483716    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2074ed-2819-4bea-a456-18987b698cca-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.483829    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.483942    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.484005    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2074ed-2819-4bea-a456-18987b698cca-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.484067    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.484117    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.488395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.484691    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"]
Nov 03 13:41:38.492433 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.492407    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.502800 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.502755    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"]
Nov 03 13:41:38.585790 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.585695    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmg55\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-kube-api-access-mmg55\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.585790 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.585748    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586031 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.585838    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586031 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.585867    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586031 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.585889    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c95\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-kube-api-access-r5c95\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586031 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.585909    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.586031 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.585950    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.586332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586083    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2074ed-2819-4bea-a456-18987b698cca-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586186    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586197    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586214    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.586332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586274    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586314    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.586652 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586460    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.586652 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586520    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586652 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586568    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.586652 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586594    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586652 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586604    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.586899 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586672    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.586899 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586695    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2074ed-2819-4bea-a456-18987b698cca-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.586899 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586757    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.587059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.586998    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.587482 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.587459    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2074ed-2819-4bea-a456-18987b698cca-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.588538 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.588515    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.588773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.588751    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2074ed-2819-4bea-a456-18987b698cca-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.597120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.597098    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c95\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-kube-api-access-r5c95\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.597252 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.597231    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.687609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.687567    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.687785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.687662    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmg55\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-kube-api-access-mmg55\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.687785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.687715    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.687785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.687756    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.687970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.687864    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.687970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.687921    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.688083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.687971    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.688083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.688036    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.688083 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.688072    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.688258 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.688234    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.688318 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.688295    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.688553 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.688533    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.688649 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.688595    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.688703 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.688687    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.690062 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.690035    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.690431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.690410    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.695994 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.695973    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.696505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.696483    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmg55\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-kube-api-access-mmg55\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:38.760571 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.760539    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:38.812887 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:38.812836    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:39.064385 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.063532    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"]
Nov 03 13:41:39.067765 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.067731    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:41:39.067864 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.067803    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:41:39.067864 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.067831    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:41:39.135928 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.135898    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"]
Nov 03 13:41:39.136575 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:41:39.136545    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7029bf0_42fd_4fbd_bc22_b93adece34c2.slice/crio-a2de094be3bd6ba7793fe703e797b03f9bb0c993a2929700bf1a179eed6ebf08 WatchSource:0}: Error finding container a2de094be3bd6ba7793fe703e797b03f9bb0c993a2929700bf1a179eed6ebf08: Status 404 returned error can't find the container with id a2de094be3bd6ba7793fe703e797b03f9bb0c993a2929700bf1a179eed6ebf08
Nov 03 13:41:39.139548 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.139517    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:41:39.139659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.139582    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:41:39.139659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.139616    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:41:39.501052 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.501009    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" event={"ID":"ee2074ed-2819-4bea-a456-18987b698cca","Type":"ContainerStarted","Data":"97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d"}
Nov 03 13:41:39.501570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.501079    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" event={"ID":"ee2074ed-2819-4bea-a456-18987b698cca","Type":"ContainerStarted","Data":"f1254a42fc895d82e2916fae9925a2a099a5a16f802fd130d4c85ce6429587a3"}
Nov 03 13:41:39.503133 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.503088    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" event={"ID":"a7029bf0-42fd-4fbd-bc22-b93adece34c2","Type":"ContainerStarted","Data":"6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb"}
Nov 03 13:41:39.503285 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.503149    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" event={"ID":"a7029bf0-42fd-4fbd-bc22-b93adece34c2","Type":"ContainerStarted","Data":"a2de094be3bd6ba7793fe703e797b03f9bb0c993a2929700bf1a179eed6ebf08"}
Nov 03 13:41:39.521562 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.521508    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podStartSLOduration=1.521493408 podStartE2EDuration="1.521493408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:39.520448298 +0000 UTC m=+1987.332116516" watchObservedRunningTime="2026-11-03 13:41:39.521493408 +0000 UTC m=+1987.333161595"
Nov 03 13:41:39.543100 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.543032    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" podStartSLOduration=1.543013456 podStartE2EDuration="1.543013456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:39.538585408 +0000 UTC m=+1987.350253620" watchObservedRunningTime="2026-11-03 13:41:39.543013456 +0000 UTC m=+1987.354681644"
Nov 03 13:41:39.761395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.761308    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:39.813587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:39.813552    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:40.018735 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:40.018640    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"]
Nov 03 13:41:40.770366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:40.769504    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 13:41:40.770366 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:40.769589    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 13:41:40.815785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:40.815747    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 13:41:40.815974 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:40.815812    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 13:41:41.764800 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:41.764758    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 13:41:41.765098 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:41.765066    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 13:41:41.813438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:41.813401    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 13:41:41.813829 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:41.813459    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 13:41:42.763775 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:42.763737    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 13:41:42.764056 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:42.764029    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 13:41:42.813739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:42.813705    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 13:41:42.814199 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:42.814162    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 13:41:43.764367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:43.763764    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 13:41:43.764367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:43.763825    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 13:41:43.814786 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:43.814749    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 13:41:43.815188 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:43.814812    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 13:41:44.776380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:44.775676    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 13:41:44.776380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:44.775740    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 13:41:44.814434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:44.813692    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 13:41:44.814434 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:44.813758    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 13:41:45.764534 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:45.764490    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 13:41:45.765134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:45.765101    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 13:41:45.822375 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:45.813322    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 13:41:45.822375 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:45.813402    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 13:41:46.762367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:46.761723    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 13:41:46.762367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:46.761788    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 13:41:46.815364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:46.814710    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 13:41:46.815364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:46.814784    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 13:41:48.206573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:48.206531    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:48.214396 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:48.214293    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:48.216239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:48.216210    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:48.761400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:48.761335    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:41:48.761599 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:48.761434    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:41:49.068933 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:49.068844    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:49.070175 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:49.070150    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:49.074167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:49.071763    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:53.738189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:53.738146    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"]
Nov 03 13:41:53.738691 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:53.738433    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy" containerID="cri-o://97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d" gracePeriod=30
Nov 03 13:41:53.750455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:53.750424    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"]
Nov 03 13:41:53.750807 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:53.750647    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy" containerID="cri-o://586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c" gracePeriod=30
Nov 03 13:41:53.755900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:53.755872    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"]
Nov 03 13:41:53.756073 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:53.756053    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy" containerID="cri-o://6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb" gracePeriod=30
Nov 03 13:41:55.048588 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.048524    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4" podUID="a1d05f68-2b0a-4410-a346-793bab7aabae" containerName="main" containerID="cri-o://6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2" gracePeriod=45
Nov 03 13:41:55.361283 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.361257    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:41:55.493239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.492803    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-tmp-dir\") pod \"a1d05f68-2b0a-4410-a346-793bab7aabae\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") "
Nov 03 13:41:55.493239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.492870    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-dshm\") pod \"a1d05f68-2b0a-4410-a346-793bab7aabae\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") "
Nov 03 13:41:55.493239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.492945    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-home\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-home\") pod \"a1d05f68-2b0a-4410-a346-793bab7aabae\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") "
Nov 03 13:41:55.493239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.493021    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-model-cache\") pod \"a1d05f68-2b0a-4410-a346-793bab7aabae\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") "
Nov 03 13:41:55.493239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.493104    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d05f68-2b0a-4410-a346-793bab7aabae-kube-api-access-vkz9x\" (UniqueName: \"kubernetes.io/projected/a1d05f68-2b0a-4410-a346-793bab7aabae-kube-api-access-vkz9x\") pod \"a1d05f68-2b0a-4410-a346-793bab7aabae\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") "
Nov 03 13:41:55.494137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.493975    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1d05f68-2b0a-4410-a346-793bab7aabae-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d05f68-2b0a-4410-a346-793bab7aabae-tls-certs\") pod \"a1d05f68-2b0a-4410-a346-793bab7aabae\" (UID: \"a1d05f68-2b0a-4410-a346-793bab7aabae\") "
Nov 03 13:41:55.498316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.498108    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d05f68-2b0a-4410-a346-793bab7aabae-tls-certs" pod "a1d05f68-2b0a-4410-a346-793bab7aabae" (UID: "a1d05f68-2b0a-4410-a346-793bab7aabae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:55.498808 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.498525    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-tmp-dir" pod "a1d05f68-2b0a-4410-a346-793bab7aabae" (UID: "a1d05f68-2b0a-4410-a346-793bab7aabae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:55.503373 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.503269    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-home" pod "a1d05f68-2b0a-4410-a346-793bab7aabae" (UID: "a1d05f68-2b0a-4410-a346-793bab7aabae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:55.503373 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.503287    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-dshm" pod "a1d05f68-2b0a-4410-a346-793bab7aabae" (UID: "a1d05f68-2b0a-4410-a346-793bab7aabae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:55.503373 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.503322    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-model-cache" pod "a1d05f68-2b0a-4410-a346-793bab7aabae" (UID: "a1d05f68-2b0a-4410-a346-793bab7aabae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:55.505129 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.505104    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d05f68-2b0a-4410-a346-793bab7aabae-kube-api-access-vkz9x" pod "a1d05f68-2b0a-4410-a346-793bab7aabae" (UID: "a1d05f68-2b0a-4410-a346-793bab7aabae"). InnerVolumeSpecName "kube-api-access-vkz9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:55.596806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.596725    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d05f68-2b0a-4410-a346-793bab7aabae-tls-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:55.596806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.596755    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-tmp-dir\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:55.596806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.596765    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-dshm\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:55.596806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.596773    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-home\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:55.596806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.596781    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d05f68-2b0a-4410-a346-793bab7aabae-model-cache\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:55.596806 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.596791    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkz9x\" (UniqueName: \"kubernetes.io/projected/a1d05f68-2b0a-4410-a346-793bab7aabae-kube-api-access-vkz9x\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:55.727418 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.727385    4447 generic.go:356] "Generic (PLEG): container finished" podID="a1d05f68-2b0a-4410-a346-793bab7aabae" containerID="6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2" exitCode=0
Nov 03 13:41:55.727590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.727431    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4" event={"ID":"a1d05f68-2b0a-4410-a346-793bab7aabae","Type":"ContainerDied","Data":"6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2"}
Nov 03 13:41:55.727590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.727469    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4" event={"ID":"a1d05f68-2b0a-4410-a346-793bab7aabae","Type":"ContainerDied","Data":"91f171c593c7e7381bcc55661207354b84c83ca613658c60a272d14d470e169e"}
Nov 03 13:41:55.727590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.727486    4447 scope.go:122] "RemoveContainer" containerID="6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2"
Nov 03 13:41:55.727590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.727500    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"
Nov 03 13:41:55.738416 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.738394    4447 scope.go:122] "RemoveContainer" containerID="6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2"
Nov 03 13:41:55.738685 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:41:55.738667    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2\": container with ID starting with 6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2 not found: ID does not exist" containerID="6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2"
Nov 03 13:41:55.738739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.738693    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2"} err="failed to get container status \"6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2\": rpc error: code = NotFound desc = could not find container \"6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2\": container with ID starting with 6b7e9621389747336c484cac0da279838239df61eac65589258a20b0d93864d2 not found: ID does not exist"
Nov 03 13:41:55.757359 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.757319    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"]
Nov 03 13:41:55.762947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:55.762923    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4"]
Nov 03 13:41:56.301193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:56.301151    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-2b8a08/sub-test-model-2b8a08-kserve-686ddf9cdc-jrdh4" podUID="a1d05f68-2b0a-4410-a346-793bab7aabae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:41:56.322276 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:56.322243    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d05f68-2b0a-4410-a346-793bab7aabae" path="/var/lib/kubelet/pods/a1d05f68-2b0a-4410-a346-793bab7aabae/volumes"
Nov 03 13:41:56.785013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:56.784973    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"]
Nov 03 13:41:56.785220 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:56.785184    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy" containerID="cri-o://720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696" gracePeriod=30
Nov 03 13:41:59.039618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.039253    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:41:59.082397 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.082331    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154174    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-envoy\") pod \"ee2074ed-2819-4bea-a456-18987b698cca\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154223    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-istio-token\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-istio-token\") pod \"ee2074ed-2819-4bea-a456-18987b698cca\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154259    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-data\") pod \"ee2074ed-2819-4bea-a456-18987b698cca\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154302    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-istio-token\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-istio-token\") pod \"5887cf56-37a7-433e-9baf-27e0ded56402\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154356    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-certs\") pod \"5887cf56-37a7-433e-9baf-27e0ded56402\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154411    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-credential-socket\") pod \"ee2074ed-2819-4bea-a456-18987b698cca\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154444    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-credential-socket\") pod \"5887cf56-37a7-433e-9baf-27e0ded56402\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154479    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-envoy\") pod \"5887cf56-37a7-433e-9baf-27e0ded56402\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154528    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-kube-api-access-r5c95\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-kube-api-access-r5c95\") pod \"ee2074ed-2819-4bea-a456-18987b698cca\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154606    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-kube-api-access-zwr5w\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-kube-api-access-zwr5w\") pod \"5887cf56-37a7-433e-9baf-27e0ded56402\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154656    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-data\") pod \"5887cf56-37a7-433e-9baf-27e0ded56402\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154774    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-certs\") pod \"ee2074ed-2819-4bea-a456-18987b698cca\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154809    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-socket\") pod \"ee2074ed-2819-4bea-a456-18987b698cca\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154878    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5887cf56-37a7-433e-9baf-27e0ded56402-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887cf56-37a7-433e-9baf-27e0ded56402-istiod-ca-cert\") pod \"5887cf56-37a7-433e-9baf-27e0ded56402\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154928    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee2074ed-2819-4bea-a456-18987b698cca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2074ed-2819-4bea-a456-18987b698cca-istiod-ca-cert\") pod \"ee2074ed-2819-4bea-a456-18987b698cca\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.154970    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5887cf56-37a7-433e-9baf-27e0ded56402-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887cf56-37a7-433e-9baf-27e0ded56402-istio-podinfo\") pod \"5887cf56-37a7-433e-9baf-27e0ded56402\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.155025    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-socket\") pod \"5887cf56-37a7-433e-9baf-27e0ded56402\" (UID: \"5887cf56-37a7-433e-9baf-27e0ded56402\") "
Nov 03 13:41:59.156095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.155110    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee2074ed-2819-4bea-a456-18987b698cca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2074ed-2819-4bea-a456-18987b698cca-istio-podinfo\") pod \"ee2074ed-2819-4bea-a456-18987b698cca\" (UID: \"ee2074ed-2819-4bea-a456-18987b698cca\") "
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.160360    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-data" pod "5887cf56-37a7-433e-9baf-27e0ded56402" (UID: "5887cf56-37a7-433e-9baf-27e0ded56402"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.160883    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5887cf56-37a7-433e-9baf-27e0ded56402-istiod-ca-cert" pod "5887cf56-37a7-433e-9baf-27e0ded56402" (UID: "5887cf56-37a7-433e-9baf-27e0ded56402"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.161059    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-socket" pod "5887cf56-37a7-433e-9baf-27e0ded56402" (UID: "5887cf56-37a7-433e-9baf-27e0ded56402"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.161095    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-socket" pod "ee2074ed-2819-4bea-a456-18987b698cca" (UID: "ee2074ed-2819-4bea-a456-18987b698cca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.161230    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-certs" pod "ee2074ed-2819-4bea-a456-18987b698cca" (UID: "ee2074ed-2819-4bea-a456-18987b698cca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.161320    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-credential-socket" pod "ee2074ed-2819-4bea-a456-18987b698cca" (UID: "ee2074ed-2819-4bea-a456-18987b698cca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.161459    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-certs" pod "5887cf56-37a7-433e-9baf-27e0ded56402" (UID: "5887cf56-37a7-433e-9baf-27e0ded56402"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.161655    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-credential-socket" pod "5887cf56-37a7-433e-9baf-27e0ded56402" (UID: "5887cf56-37a7-433e-9baf-27e0ded56402"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.164849    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-kube-api-access-r5c95" pod "ee2074ed-2819-4bea-a456-18987b698cca" (UID: "ee2074ed-2819-4bea-a456-18987b698cca"). InnerVolumeSpecName "kube-api-access-r5c95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.165175    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-data" pod "ee2074ed-2819-4bea-a456-18987b698cca" (UID: "ee2074ed-2819-4bea-a456-18987b698cca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.165499    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-envoy" pod "5887cf56-37a7-433e-9baf-27e0ded56402" (UID: "5887cf56-37a7-433e-9baf-27e0ded56402"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.166526    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2074ed-2819-4bea-a456-18987b698cca-istiod-ca-cert" pod "ee2074ed-2819-4bea-a456-18987b698cca" (UID: "ee2074ed-2819-4bea-a456-18987b698cca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:59.166587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.166551    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-istio-token" pod "ee2074ed-2819-4bea-a456-18987b698cca" (UID: "ee2074ed-2819-4bea-a456-18987b698cca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.166955    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c95\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-kube-api-access-r5c95\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.166982    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.166998    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167012    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167025    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887cf56-37a7-433e-9baf-27e0ded56402-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167037    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2074ed-2819-4bea-a456-18987b698cca-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167051    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167065    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2074ed-2819-4bea-a456-18987b698cca-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167079    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167142    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167157    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167171    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167186    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887cf56-37a7-433e-9baf-27e0ded56402-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167312    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-envoy" pod "ee2074ed-2819-4bea-a456-18987b698cca" (UID: "ee2074ed-2819-4bea-a456-18987b698cca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.167650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.167510    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-kube-api-access-zwr5w" pod "5887cf56-37a7-433e-9baf-27e0ded56402" (UID: "5887cf56-37a7-433e-9baf-27e0ded56402"). InnerVolumeSpecName "kube-api-access-zwr5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.177544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.175592    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee2074ed-2819-4bea-a456-18987b698cca-istio-podinfo" pod "ee2074ed-2819-4bea-a456-18987b698cca" (UID: "ee2074ed-2819-4bea-a456-18987b698cca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:41:59.177544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.177240    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5887cf56-37a7-433e-9baf-27e0ded56402-istio-podinfo" pod "5887cf56-37a7-433e-9baf-27e0ded56402" (UID: "5887cf56-37a7-433e-9baf-27e0ded56402"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:41:59.177544 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.177499    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-istio-token" pod "5887cf56-37a7-433e-9baf-27e0ded56402" (UID: "5887cf56-37a7-433e-9baf-27e0ded56402"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.180162 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.180138    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:59.268318 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268242    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-podinfo\") pod \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") "
Nov 03 13:41:59.268318 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268286    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-credential-socket\") pod \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") "
Nov 03 13:41:59.268559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268404    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-data\") pod \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") "
Nov 03 13:41:59.268559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268456    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-envoy\") pod \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") "
Nov 03 13:41:59.268559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268515    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-kube-api-access-mmg55\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-kube-api-access-mmg55\") pod \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") "
Nov 03 13:41:59.268559 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268541    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istiod-ca-cert\") pod \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") "
Nov 03 13:41:59.268787 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268605    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-certs\") pod \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") "
Nov 03 13:41:59.268787 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268666    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-credential-socket" pod "a7029bf0-42fd-4fbd-bc22-b93adece34c2" (UID: "a7029bf0-42fd-4fbd-bc22-b93adece34c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.268787 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268702    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-token\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-token\") pod \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") "
Nov 03 13:41:59.268787 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268722    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-data" pod "a7029bf0-42fd-4fbd-bc22-b93adece34c2" (UID: "a7029bf0-42fd-4fbd-bc22-b93adece34c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.269065 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.268812    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-socket\") pod \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\" (UID: \"a7029bf0-42fd-4fbd-bc22-b93adece34c2\") "
Nov 03 13:41:59.269555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269070    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istiod-ca-cert" pod "a7029bf0-42fd-4fbd-bc22-b93adece34c2" (UID: "a7029bf0-42fd-4fbd-bc22-b93adece34c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:59.269555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269493    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.269555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269517    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwr5w\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-kube-api-access-zwr5w\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.269555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269533    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.269555 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269546    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.269917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269560    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887cf56-37a7-433e-9baf-27e0ded56402-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.269917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269575    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2074ed-2819-4bea-a456-18987b698cca-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.269917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269589    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2074ed-2819-4bea-a456-18987b698cca-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.269917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269647    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5887cf56-37a7-433e-9baf-27e0ded56402-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.269917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269813    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-certs" pod "a7029bf0-42fd-4fbd-bc22-b93adece34c2" (UID: "a7029bf0-42fd-4fbd-bc22-b93adece34c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.270093 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.269920    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-socket" pod "a7029bf0-42fd-4fbd-bc22-b93adece34c2" (UID: "a7029bf0-42fd-4fbd-bc22-b93adece34c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.271360 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.271322    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-kube-api-access-mmg55" pod "a7029bf0-42fd-4fbd-bc22-b93adece34c2" (UID: "a7029bf0-42fd-4fbd-bc22-b93adece34c2"). InnerVolumeSpecName "kube-api-access-mmg55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.271632 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.271600    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-envoy" pod "a7029bf0-42fd-4fbd-bc22-b93adece34c2" (UID: "a7029bf0-42fd-4fbd-bc22-b93adece34c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.271748 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.271732    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-token" pod "a7029bf0-42fd-4fbd-bc22-b93adece34c2" (UID: "a7029bf0-42fd-4fbd-bc22-b93adece34c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.273235 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.273147    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-podinfo" pod "a7029bf0-42fd-4fbd-bc22-b93adece34c2" (UID: "a7029bf0-42fd-4fbd-bc22-b93adece34c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:41:59.371189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.371150    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.371189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.371188    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmg55\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-kube-api-access-mmg55\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.371429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.371199    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.371429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.371208    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.371429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.371217    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7029bf0-42fd-4fbd-bc22-b93adece34c2-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.371429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.371226    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7029bf0-42fd-4fbd-bc22-b93adece34c2-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.759456 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.759422    4447 generic.go:356] "Generic (PLEG): container finished" podID="5887cf56-37a7-433e-9baf-27e0ded56402" containerID="586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c" exitCode=0
Nov 03 13:41:59.759659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.759487    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"
Nov 03 13:41:59.759659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.759507    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" event={"ID":"5887cf56-37a7-433e-9baf-27e0ded56402","Type":"ContainerDied","Data":"586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c"}
Nov 03 13:41:59.759659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.759573    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq" event={"ID":"5887cf56-37a7-433e-9baf-27e0ded56402","Type":"ContainerDied","Data":"ade7825e367ffae1af4f03c67b3538588be403fe227f277f85ff56106db4e673"}
Nov 03 13:41:59.759659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.759602    4447 scope.go:122] "RemoveContainer" containerID="586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c"
Nov 03 13:41:59.761156 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.761130    4447 generic.go:356] "Generic (PLEG): container finished" podID="ee2074ed-2819-4bea-a456-18987b698cca" containerID="97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d" exitCode=0
Nov 03 13:41:59.761286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.761184    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"
Nov 03 13:41:59.761286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.761208    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" event={"ID":"ee2074ed-2819-4bea-a456-18987b698cca","Type":"ContainerDied","Data":"97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d"}
Nov 03 13:41:59.761286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.761269    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x" event={"ID":"ee2074ed-2819-4bea-a456-18987b698cca","Type":"ContainerDied","Data":"f1254a42fc895d82e2916fae9925a2a099a5a16f802fd130d4c85ce6429587a3"}
Nov 03 13:41:59.764489 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.764465    4447 generic.go:356] "Generic (PLEG): container finished" podID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerID="6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb" exitCode=0
Nov 03 13:41:59.764586 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.764499    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" event={"ID":"a7029bf0-42fd-4fbd-bc22-b93adece34c2","Type":"ContainerDied","Data":"6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb"}
Nov 03 13:41:59.764586 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.764538    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm" event={"ID":"a7029bf0-42fd-4fbd-bc22-b93adece34c2","Type":"ContainerDied","Data":"a2de094be3bd6ba7793fe703e797b03f9bb0c993a2929700bf1a179eed6ebf08"}
Nov 03 13:41:59.764714 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.764602    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"
Nov 03 13:41:59.796384 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.796356    4447 scope.go:122] "RemoveContainer" containerID="586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c"
Nov 03 13:41:59.796746 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:41:59.796724    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c\": container with ID starting with 586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c not found: ID does not exist" containerID="586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c"
Nov 03 13:41:59.796823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.796754    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c"} err="failed to get container status \"586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c\": rpc error: code = NotFound desc = could not find container \"586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c\": container with ID starting with 586bb25bc7c0fa0188fb7dd14be65d35d1d506b4e7155d2b31abdb26a74cb26c not found: ID does not exist"
Nov 03 13:41:59.796823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.796772    4447 scope.go:122] "RemoveContainer" containerID="97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d"
Nov 03 13:41:59.806195 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.806177    4447 scope.go:122] "RemoveContainer" containerID="97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d"
Nov 03 13:41:59.806447 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:41:59.806429    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d\": container with ID starting with 97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d not found: ID does not exist" containerID="97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d"
Nov 03 13:41:59.806510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.806455    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d"} err="failed to get container status \"97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d\": rpc error: code = NotFound desc = could not find container \"97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d\": container with ID starting with 97cab37297ede84bc47a7e217dbc34964c5efff11e6371c6a7f4b806e74b205d not found: ID does not exist"
Nov 03 13:41:59.806510 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.806472    4447 scope.go:122] "RemoveContainer" containerID="6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb"
Nov 03 13:41:59.814025 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.814000    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"]
Nov 03 13:41:59.817214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.817182    4447 scope.go:122] "RemoveContainer" containerID="6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb"
Nov 03 13:41:59.817553 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:41:59.817533    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb\": container with ID starting with 6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb not found: ID does not exist" containerID="6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb"
Nov 03 13:41:59.817637 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.817564    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb"} err="failed to get container status \"6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb\": rpc error: code = NotFound desc = could not find container \"6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb\": container with ID starting with 6fa44c787f7268bf759bf9334110e4470dd5e9e16a3fee5e7c564952137b24cb not found: ID does not exist"
Nov 03 13:41:59.819642 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.819620    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-g229x"]
Nov 03 13:41:59.834594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.834572    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"]
Nov 03 13:41:59.841028 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.840998    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwqbq"]
Nov 03 13:41:59.858191 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.858165    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"]
Nov 03 13:41:59.861927 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:41:59.861903    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v58pm"]
Nov 03 13:42:00.320158 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:00.320122    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" path="/var/lib/kubelet/pods/5887cf56-37a7-433e-9baf-27e0ded56402/volumes"
Nov 03 13:42:00.320648 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:00.320632    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" path="/var/lib/kubelet/pods/a7029bf0-42fd-4fbd-bc22-b93adece34c2/volumes"
Nov 03 13:42:00.321073 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:00.321061    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee2074ed-2819-4bea-a456-18987b698cca" path="/var/lib/kubelet/pods/ee2074ed-2819-4bea-a456-18987b698cca/volumes"
Nov 03 13:42:01.620757 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:01.620718    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:42:01.621147 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:01.620784    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:42:02.051970 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.051942    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.210487    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-socket\") pod \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") "
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.210612    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-podinfo\") pod \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") "
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.210650    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-data\") pod \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") "
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.210681    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-kube-api-access-cr29g\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-kube-api-access-cr29g\") pod \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") "
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.210724    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istiod-ca-cert\") pod \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") "
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.210774    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-envoy\") pod \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") "
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.210834    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-token\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-token\") pod \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") "
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.210947    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-certs\") pod \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") "
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.211032    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-credential-socket\") pod \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\" (UID: \"8e7290ec-02e1-41f4-8c59-18aa7f1130dc\") "
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.212840    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-data" pod "8e7290ec-02e1-41f4-8c59-18aa7f1130dc" (UID: "8e7290ec-02e1-41f4-8c59-18aa7f1130dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.212929    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-socket" pod "8e7290ec-02e1-41f4-8c59-18aa7f1130dc" (UID: "8e7290ec-02e1-41f4-8c59-18aa7f1130dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.212976    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-certs" pod "8e7290ec-02e1-41f4-8c59-18aa7f1130dc" (UID: "8e7290ec-02e1-41f4-8c59-18aa7f1130dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:02.213160 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.213093    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-credential-socket" pod "8e7290ec-02e1-41f4-8c59-18aa7f1130dc" (UID: "8e7290ec-02e1-41f4-8c59-18aa7f1130dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:02.214335 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.214312    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istiod-ca-cert" pod "8e7290ec-02e1-41f4-8c59-18aa7f1130dc" (UID: "8e7290ec-02e1-41f4-8c59-18aa7f1130dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:02.214889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.214861    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-podinfo" pod "8e7290ec-02e1-41f4-8c59-18aa7f1130dc" (UID: "8e7290ec-02e1-41f4-8c59-18aa7f1130dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:42:02.215570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.215531    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-token" pod "8e7290ec-02e1-41f4-8c59-18aa7f1130dc" (UID: "8e7290ec-02e1-41f4-8c59-18aa7f1130dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:02.216866 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.216841    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-kube-api-access-cr29g" pod "8e7290ec-02e1-41f4-8c59-18aa7f1130dc" (UID: "8e7290ec-02e1-41f4-8c59-18aa7f1130dc"). InnerVolumeSpecName "kube-api-access-cr29g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:02.218266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.218197    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-envoy" pod "8e7290ec-02e1-41f4-8c59-18aa7f1130dc" (UID: "8e7290ec-02e1-41f4-8c59-18aa7f1130dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:02.313214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.313174    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:02.313214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.313214    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:02.313497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.313233    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:02.313497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.313248    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr29g\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-kube-api-access-cr29g\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:02.313497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.313261    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:02.313497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.313276    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:02.313497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.313289    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:02.313497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.313302    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:02.313497 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.313315    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7290ec-02e1-41f4-8c59-18aa7f1130dc-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:02.817201 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.817166    4447 generic.go:356] "Generic (PLEG): container finished" podID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerID="720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696" exitCode=0
Nov 03 13:42:02.817730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.817216    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"
Nov 03 13:42:02.817730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.817253    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" event={"ID":"8e7290ec-02e1-41f4-8c59-18aa7f1130dc","Type":"ContainerDied","Data":"720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696"}
Nov 03 13:42:02.817730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.817308    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj" event={"ID":"8e7290ec-02e1-41f4-8c59-18aa7f1130dc","Type":"ContainerDied","Data":"4463ee5db4e523653d2b91e574cc75e9d71cb72edeef9f0ee3c4fd5e81c52427"}
Nov 03 13:42:02.817730 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.817337    4447 scope.go:122] "RemoveContainer" containerID="720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696"
Nov 03 13:42:02.827498 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.827476    4447 scope.go:122] "RemoveContainer" containerID="720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696"
Nov 03 13:42:02.827799 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:42:02.827780    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696\": container with ID starting with 720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696 not found: ID does not exist" containerID="720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696"
Nov 03 13:42:02.827869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.827813    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696"} err="failed to get container status \"720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696\": rpc error: code = NotFound desc = could not find container \"720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696\": container with ID starting with 720fd01f262145f4fc858c20c0470f3edf36f6f49cd01c287308781988213696 not found: ID does not exist"
Nov 03 13:42:02.834599 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.834572    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"]
Nov 03 13:42:02.840285 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:02.840252    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-9tqtj"]
Nov 03 13:42:03.415777 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:42:03.415735    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:42:04.323198 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:04.323163    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" path="/var/lib/kubelet/pods/8e7290ec-02e1-41f4-8c59-18aa7f1130dc/volumes"
Nov 03 13:42:27.968718 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:27.968641    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"]
Nov 03 13:42:27.982327 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:27.982292    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"]
Nov 03 13:42:32.590021 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:32.589948    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"]
Nov 03 13:42:32.591006 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:32.590952    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65" podUID="f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" containerName="maas-api" containerID="cri-o://7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54" gracePeriod=30
Nov 03 13:42:32.969196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:32.969052    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz" podUID="21a6efbf-968c-4e5d-93b3-a62eaae802d6" containerName="payload-processing" containerID="cri-o://6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd" gracePeriod=25
Nov 03 13:42:32.973752 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:32.973727    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:42:32.982595 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:32.982546    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9" podUID="47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" containerName="payload-pre-processing" containerID="cri-o://29c9a42920479721a116b273eac9bf5e995662db831f2532e6026922fe1efc5d" gracePeriod=25
Nov 03 13:42:33.087573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.087534    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-maas-api-tls\") pod \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") "
Nov 03 13:42:33.087774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.087643    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-kube-api-access-8nklk\" (UniqueName: \"kubernetes.io/projected/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-kube-api-access-8nklk\") pod \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") "
Nov 03 13:42:33.087774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.087723    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-metrics-certs\") pod \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") "
Nov 03 13:42:33.087774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.087762    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-openshift-service-ca\") pod \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\" (UID: \"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e\") "
Nov 03 13:42:33.088199 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.088167    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-openshift-service-ca" pod "f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" (UID: "f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:33.089852 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.089823    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-maas-api-tls" pod "f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" (UID: "f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:42:33.089973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.089858    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-kube-api-access-8nklk" pod "f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" (UID: "f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e"). InnerVolumeSpecName "kube-api-access-8nklk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:33.089973 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.089883    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-metrics-certs" pod "f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" (UID: "f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:42:33.158438 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.158390    4447 generic.go:356] "Generic (PLEG): container finished" podID="f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" containerID="7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54" exitCode=0
Nov 03 13:42:33.158627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.158458    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"
Nov 03 13:42:33.158627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.158459    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65" event={"ID":"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e","Type":"ContainerDied","Data":"7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54"}
Nov 03 13:42:33.158627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.158536    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65" event={"ID":"f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e","Type":"ContainerDied","Data":"6da4b30eaa328b4ecbfa13675ae757bab9b0cb13b218ec920f67242bec946e0b"}
Nov 03 13:42:33.158627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.158559    4447 scope.go:122] "RemoveContainer" containerID="7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54"
Nov 03 13:42:33.177726 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.177694    4447 scope.go:122] "RemoveContainer" containerID="7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54"
Nov 03 13:42:33.178142 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:42:33.178114    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54\": container with ID starting with 7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54 not found: ID does not exist" containerID="7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54"
Nov 03 13:42:33.178233 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.178154    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54"} err="failed to get container status \"7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54\": rpc error: code = NotFound desc = could not find container \"7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54\": container with ID starting with 7f2027b0e0e2d38a89f57e9e7419a40f5ba7b2835a7deee416f29ad0ef1bac54 not found: ID does not exist"
Nov 03 13:42:33.189275 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.189246    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:33.189275 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.189274    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:33.189491 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.189283    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:33.189491 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.189293    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nklk\" (UniqueName: \"kubernetes.io/projected/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e-kube-api-access-8nklk\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:33.199222 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.199037    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"]
Nov 03 13:42:33.205838 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.205808    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a724a6-cd5d556fc-97v65"]
Nov 03 13:42:33.412905 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:42:33.412861    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:42:33.435477 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:33.435445    4447 scope.go:122] "RemoveContainer" containerID="c609794f0083036ec9928259fd4641f1b6fe4b325af4ecf996fbd8eb41300a16"
Nov 03 13:42:34.319548 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.319512    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" path="/var/lib/kubelet/pods/f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e/volumes"
Nov 03 13:42:34.394399 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.394370    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"]
Nov 03 13:42:34.394987 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.394972    4447 cpu_manager.go:414] "containerMap: removing container" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy"
Nov 03 13:42:34.395045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.394989    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy"
Nov 03 13:42:34.395045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395003    4447 cpu_manager.go:414] "containerMap: removing container" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy"
Nov 03 13:42:34.395045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395008    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy"
Nov 03 13:42:34.395045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395026    4447 cpu_manager.go:414] "containerMap: removing container" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy"
Nov 03 13:42:34.395045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395031    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy"
Nov 03 13:42:34.395045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395038    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" containerName="maas-api"
Nov 03 13:42:34.395045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395043    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" containerName="maas-api"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395051    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395055    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395074    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a1d05f68-2b0a-4410-a346-793bab7aabae" containerName="main"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395079    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d05f68-2b0a-4410-a346-793bab7aabae" containerName="main"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395162    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7290ec-02e1-41f4-8c59-18aa7f1130dc" containerName="istio-proxy"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395170    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f40f2b-a4bd-4aae-8e18-b88afd5e5c7e" containerName="maas-api"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395177    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7029bf0-42fd-4fbd-bc22-b93adece34c2" containerName="istio-proxy"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395187    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d05f68-2b0a-4410-a346-793bab7aabae" containerName="main"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395196    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee2074ed-2819-4bea-a456-18987b698cca" containerName="istio-proxy"
Nov 03 13:42:34.395292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.395203    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="5887cf56-37a7-433e-9baf-27e0ded56402" containerName="istio-proxy"
Nov 03 13:42:34.400625 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.400601    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.418529 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.418504    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"]
Nov 03 13:42:34.513514 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.512409    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.513514 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.512472    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.513514 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.512520    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.514151 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.514111    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzj5z\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-kube-api-access-xzj5z\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.514298 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.514197    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.514298 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.514254    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.514480 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.514322    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.514480 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.514465    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.514808 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.514543    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616105    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616196 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616167    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616519 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616235    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616519 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616283    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616519 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616332    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616519 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616386    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616519 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616426    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616519 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616493    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzj5z\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-kube-api-access-xzj5z\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616550    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616672    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.616889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616745    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.617041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616888    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.617041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.616943    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.617265 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.617241    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.618847 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.618819    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.619218 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.619194    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.624649 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.624627    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.624773 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.624626    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzj5z\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-kube-api-access-xzj5z\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:34.721197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:34.721164    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:35.009895 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:35.009865    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"]
Nov 03 13:42:35.011862 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:42:35.011838    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b45341_138a_4e2e_a5bd_85fc060d7e17.slice/crio-2bc22f08f7cb162b2b98e6a757d1bec3a9b94ad7c2516214a3602df46dc70bfe WatchSource:0}: Error finding container 2bc22f08f7cb162b2b98e6a757d1bec3a9b94ad7c2516214a3602df46dc70bfe: Status 404 returned error can't find the container with id 2bc22f08f7cb162b2b98e6a757d1bec3a9b94ad7c2516214a3602df46dc70bfe
Nov 03 13:42:35.014562 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:35.014528    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:42:35.014673 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:35.014593    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:42:35.014673 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:35.014621    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:42:35.175399 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:35.174114    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" event={"ID":"d6b45341-138a-4e2e-a5bd-85fc060d7e17","Type":"ContainerStarted","Data":"8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab"}
Nov 03 13:42:35.175399 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:35.174184    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" event={"ID":"d6b45341-138a-4e2e-a5bd-85fc060d7e17","Type":"ContainerStarted","Data":"2bc22f08f7cb162b2b98e6a757d1bec3a9b94ad7c2516214a3602df46dc70bfe"}
Nov 03 13:42:35.202354 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:35.202284    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" podStartSLOduration=1.202265578 podStartE2EDuration="1.202265578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:35.198388428 +0000 UTC m=+2043.010056638" watchObservedRunningTime="2026-11-03 13:42:35.202265578 +0000 UTC m=+2043.013933770"
Nov 03 13:42:35.721278 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:35.721233    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:36.231536 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:36.231498    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9" podUID="47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:42:36.231718 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:36.231498    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz" podUID="21a6efbf-968c-4e5d-93b3-a62eaae802d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:42:36.599100 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:36.599007    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"]
Nov 03 13:42:36.599375 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:36.599338    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"]
Nov 03 13:42:36.599797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:36.599761    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy" containerID="cri-o://cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd" gracePeriod=30
Nov 03 13:42:36.600176 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:36.600152    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy" containerID="cri-o://01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416" gracePeriod=30
Nov 03 13:42:36.790365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:36.785499    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 13:42:36.790365 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:36.785563    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 13:42:37.722171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:37.722128    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 13:42:37.722425 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:37.722204    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 13:42:38.722272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:38.722236    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 13:42:38.722693 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:38.722300    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 13:42:39.721770 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:39.721731    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 13:42:39.721950 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:39.721795    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 13:42:40.721959 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:40.721921    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 13:42:40.722369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:40.721997    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 13:42:40.859573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:40.859521    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:42:40.859925 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:40.859895    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:42:41.721965 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.721935    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 13:42:41.722086 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.721999    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 13:42:41.723724 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.723698    4447 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 13:42:41.723849 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.723748    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 13:42:41.907182 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.907150    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:42:41.950699 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.950668    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:42:41.998056 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998023    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-token\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-token\") pod \"aff4deac-8242-44bb-9b15-f7f57d2f7145\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") "
Nov 03 13:42:41.998240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998075    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-kube-api-access-4sdtq\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-kube-api-access-4sdtq\") pod \"e73dcbee-73b7-4bd7-9254-09db37d787c9\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") "
Nov 03 13:42:41.998240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998125    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-socket\") pod \"e73dcbee-73b7-4bd7-9254-09db37d787c9\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") "
Nov 03 13:42:41.998240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998166    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-data\") pod \"aff4deac-8242-44bb-9b15-f7f57d2f7145\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") "
Nov 03 13:42:41.998240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998197    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-token\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-token\") pod \"e73dcbee-73b7-4bd7-9254-09db37d787c9\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") "
Nov 03 13:42:41.998240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998238    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-kube-api-access-z5w7b\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-kube-api-access-z5w7b\") pod \"aff4deac-8242-44bb-9b15-f7f57d2f7145\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") "
Nov 03 13:42:41.998594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998272    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-credential-socket\") pod \"e73dcbee-73b7-4bd7-9254-09db37d787c9\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") "
Nov 03 13:42:41.998594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998315    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-certs\") pod \"aff4deac-8242-44bb-9b15-f7f57d2f7145\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") "
Nov 03 13:42:41.998594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998363    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-certs\") pod \"e73dcbee-73b7-4bd7-9254-09db37d787c9\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") "
Nov 03 13:42:41.998594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998413    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73dcbee-73b7-4bd7-9254-09db37d787c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73dcbee-73b7-4bd7-9254-09db37d787c9-istiod-ca-cert\") pod \"e73dcbee-73b7-4bd7-9254-09db37d787c9\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") "
Nov 03 13:42:41.998594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998457    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-socket\") pod \"aff4deac-8242-44bb-9b15-f7f57d2f7145\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") "
Nov 03 13:42:41.998594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998527    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff4deac-8242-44bb-9b15-f7f57d2f7145-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4deac-8242-44bb-9b15-f7f57d2f7145-istiod-ca-cert\") pod \"aff4deac-8242-44bb-9b15-f7f57d2f7145\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") "
Nov 03 13:42:41.998594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998557    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-podinfo\") pod \"e73dcbee-73b7-4bd7-9254-09db37d787c9\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") "
Nov 03 13:42:41.998594 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998584    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-envoy\") pod \"aff4deac-8242-44bb-9b15-f7f57d2f7145\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") "
Nov 03 13:42:41.999016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998667    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-envoy\") pod \"e73dcbee-73b7-4bd7-9254-09db37d787c9\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") "
Nov 03 13:42:41.999016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998755    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-podinfo\") pod \"aff4deac-8242-44bb-9b15-f7f57d2f7145\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") "
Nov 03 13:42:41.999016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998789    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-credential-socket\") pod \"aff4deac-8242-44bb-9b15-f7f57d2f7145\" (UID: \"aff4deac-8242-44bb-9b15-f7f57d2f7145\") "
Nov 03 13:42:41.999016 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.998830    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-data\") pod \"e73dcbee-73b7-4bd7-9254-09db37d787c9\" (UID: \"e73dcbee-73b7-4bd7-9254-09db37d787c9\") "
Nov 03 13:42:41.999229 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999074    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-credential-socket" pod "e73dcbee-73b7-4bd7-9254-09db37d787c9" (UID: "e73dcbee-73b7-4bd7-9254-09db37d787c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:41.999229 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999083    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-data" pod "aff4deac-8242-44bb-9b15-f7f57d2f7145" (UID: "aff4deac-8242-44bb-9b15-f7f57d2f7145"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:41.999354 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999264    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-data" pod "e73dcbee-73b7-4bd7-9254-09db37d787c9" (UID: "e73dcbee-73b7-4bd7-9254-09db37d787c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:41.999354 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999271    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-certs" pod "aff4deac-8242-44bb-9b15-f7f57d2f7145" (UID: "aff4deac-8242-44bb-9b15-f7f57d2f7145"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:41.999476 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999444    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:41.999476 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999018    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-socket" pod "e73dcbee-73b7-4bd7-9254-09db37d787c9" (UID: "e73dcbee-73b7-4bd7-9254-09db37d787c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:41.999476 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999466    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:41.999637 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999482    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:41.999637 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999486    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-certs" pod "e73dcbee-73b7-4bd7-9254-09db37d787c9" (UID: "e73dcbee-73b7-4bd7-9254-09db37d787c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:41.999637 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999496    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:41.999802 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999700    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-socket" pod "aff4deac-8242-44bb-9b15-f7f57d2f7145" (UID: "aff4deac-8242-44bb-9b15-f7f57d2f7145"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.000328 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:41.999984    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73dcbee-73b7-4bd7-9254-09db37d787c9-istiod-ca-cert" pod "e73dcbee-73b7-4bd7-9254-09db37d787c9" (UID: "e73dcbee-73b7-4bd7-9254-09db37d787c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:42.000328 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.000199    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff4deac-8242-44bb-9b15-f7f57d2f7145-istiod-ca-cert" pod "aff4deac-8242-44bb-9b15-f7f57d2f7145" (UID: "aff4deac-8242-44bb-9b15-f7f57d2f7145"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:42.000570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.000537    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-credential-socket" pod "aff4deac-8242-44bb-9b15-f7f57d2f7145" (UID: "aff4deac-8242-44bb-9b15-f7f57d2f7145"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.002891 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.002868    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-podinfo" pod "aff4deac-8242-44bb-9b15-f7f57d2f7145" (UID: "aff4deac-8242-44bb-9b15-f7f57d2f7145"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:42:42.004320 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.004299    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-envoy" pod "aff4deac-8242-44bb-9b15-f7f57d2f7145" (UID: "aff4deac-8242-44bb-9b15-f7f57d2f7145"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.004447 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.004385    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-kube-api-access-4sdtq" pod "e73dcbee-73b7-4bd7-9254-09db37d787c9" (UID: "e73dcbee-73b7-4bd7-9254-09db37d787c9"). InnerVolumeSpecName "kube-api-access-4sdtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:42.004447 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.004403    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-kube-api-access-z5w7b" pod "aff4deac-8242-44bb-9b15-f7f57d2f7145" (UID: "aff4deac-8242-44bb-9b15-f7f57d2f7145"). InnerVolumeSpecName "kube-api-access-z5w7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:42.004447 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.004422    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-token" pod "e73dcbee-73b7-4bd7-9254-09db37d787c9" (UID: "e73dcbee-73b7-4bd7-9254-09db37d787c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:42.004618 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.004541    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-token" pod "aff4deac-8242-44bb-9b15-f7f57d2f7145" (UID: "aff4deac-8242-44bb-9b15-f7f57d2f7145"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:42.004684 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.004662    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-envoy" pod "e73dcbee-73b7-4bd7-9254-09db37d787c9" (UID: "e73dcbee-73b7-4bd7-9254-09db37d787c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.006264 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.006241    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-podinfo" pod "e73dcbee-73b7-4bd7-9254-09db37d787c9" (UID: "e73dcbee-73b7-4bd7-9254-09db37d787c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100376    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sdtq\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-kube-api-access-4sdtq\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100412    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100429    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100446    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5w7b\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-kube-api-access-z5w7b\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100463    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100476    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73dcbee-73b7-4bd7-9254-09db37d787c9-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100488    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100502    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4deac-8242-44bb-9b15-f7f57d2f7145-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100514    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100540    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100554    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73dcbee-73b7-4bd7-9254-09db37d787c9-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100567    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100582    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4deac-8242-44bb-9b15-f7f57d2f7145-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.100619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.100595    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff4deac-8242-44bb-9b15-f7f57d2f7145-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.242877 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.242839    4447 generic.go:356] "Generic (PLEG): container finished" podID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerID="01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416" exitCode=0
Nov 03 13:42:42.243133 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.243118    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" event={"ID":"e73dcbee-73b7-4bd7-9254-09db37d787c9","Type":"ContainerDied","Data":"01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416"}
Nov 03 13:42:42.243217 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.243189    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw" event={"ID":"e73dcbee-73b7-4bd7-9254-09db37d787c9","Type":"ContainerDied","Data":"734ec0972a32f0c2a412e9a6ba7cb9b7caff2afb66b375394b3c1715737e8423"}
Nov 03 13:42:42.243309 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.243218    4447 scope.go:122] "RemoveContainer" containerID="01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416"
Nov 03 13:42:42.243309 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.243272    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"
Nov 03 13:42:42.247301 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.247270    4447 generic.go:356] "Generic (PLEG): container finished" podID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerID="cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd" exitCode=0
Nov 03 13:42:42.247485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.247336    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" event={"ID":"aff4deac-8242-44bb-9b15-f7f57d2f7145","Type":"ContainerDied","Data":"cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd"}
Nov 03 13:42:42.247485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.247339    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"
Nov 03 13:42:42.247485 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.247399    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9" event={"ID":"aff4deac-8242-44bb-9b15-f7f57d2f7145","Type":"ContainerDied","Data":"6a9322390f80bf12ab0bab336f0816fe828ec1b25225f9ffc0fe25247e157540"}
Nov 03 13:42:42.259413 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.259394    4447 scope.go:122] "RemoveContainer" containerID="01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416"
Nov 03 13:42:42.259876 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:42:42.259843    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416\": container with ID starting with 01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416 not found: ID does not exist" containerID="01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416"
Nov 03 13:42:42.259981 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.259887    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416"} err="failed to get container status \"01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416\": rpc error: code = NotFound desc = could not find container \"01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416\": container with ID starting with 01aa1be4cfd0385067c04eab1613a73a35be31bcbc43bb025959c11cc8448416 not found: ID does not exist"
Nov 03 13:42:42.259981 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.259911    4447 scope.go:122] "RemoveContainer" containerID="cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd"
Nov 03 13:42:42.274534 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.274510    4447 scope.go:122] "RemoveContainer" containerID="cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd"
Nov 03 13:42:42.274866 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:42:42.274848    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd\": container with ID starting with cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd not found: ID does not exist" containerID="cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd"
Nov 03 13:42:42.274954 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.274872    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd"} err="failed to get container status \"cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd\": rpc error: code = NotFound desc = could not find container \"cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd\": container with ID starting with cbf748a3de39c91ff680868c8514dece3fee7a6a95c463180f765687503137dd not found: ID does not exist"
Nov 03 13:42:42.279619 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.279598    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"]
Nov 03 13:42:42.284847 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.284825    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-pt9kw"]
Nov 03 13:42:42.293122 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.293086    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"]
Nov 03 13:42:42.298233 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.298212    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-tl2n9"]
Nov 03 13:42:42.322107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.322069    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" path="/var/lib/kubelet/pods/aff4deac-8242-44bb-9b15-f7f57d2f7145/volumes"
Nov 03 13:42:42.322602 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:42.322583    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" path="/var/lib/kubelet/pods/e73dcbee-73b7-4bd7-9254-09db37d787c9/volumes"
Nov 03 13:42:43.770371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:43.763489    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:42:43.770371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:43.763559    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:42:43.878453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:43.878413    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:43.880274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:43.879508    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:43.880274 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:43.880255    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:42:44.978374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:44.978309    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"]
Nov 03 13:42:44.997045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:44.997003    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"]
Nov 03 13:42:46.239398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.232559    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9" podUID="47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:42:46.239398 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.235956    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz" podUID="21a6efbf-968c-4e5d-93b3-a62eaae802d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:42:46.304676 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.304642    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"]
Nov 03 13:42:46.304901 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.304865    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb" podUID="ea5d6076-2ccf-44c4-9eef-18efb41051b8" containerName="maas-api" containerID="cri-o://72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d" gracePeriod=30
Nov 03 13:42:46.696535 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.696509    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:42:46.794358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.794312    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-metrics-certs\") pod \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") "
Nov 03 13:42:46.794505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.794395    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-maas-api-tls\") pod \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") "
Nov 03 13:42:46.794505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.794440    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5d6076-2ccf-44c4-9eef-18efb41051b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d6076-2ccf-44c4-9eef-18efb41051b8-openshift-service-ca\") pod \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") "
Nov 03 13:42:46.794605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.794558    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5d6076-2ccf-44c4-9eef-18efb41051b8-kube-api-access-5gsjc\" (UniqueName: \"kubernetes.io/projected/ea5d6076-2ccf-44c4-9eef-18efb41051b8-kube-api-access-5gsjc\") pod \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\" (UID: \"ea5d6076-2ccf-44c4-9eef-18efb41051b8\") "
Nov 03 13:42:46.794868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.794840    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5d6076-2ccf-44c4-9eef-18efb41051b8-openshift-service-ca" pod "ea5d6076-2ccf-44c4-9eef-18efb41051b8" (UID: "ea5d6076-2ccf-44c4-9eef-18efb41051b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:46.795048 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.795030    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5d6076-2ccf-44c4-9eef-18efb41051b8-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:46.796587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.796553    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-metrics-certs" pod "ea5d6076-2ccf-44c4-9eef-18efb41051b8" (UID: "ea5d6076-2ccf-44c4-9eef-18efb41051b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:42:46.796702 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.796603    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-maas-api-tls" pod "ea5d6076-2ccf-44c4-9eef-18efb41051b8" (UID: "ea5d6076-2ccf-44c4-9eef-18efb41051b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:42:46.796702 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.796672    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5d6076-2ccf-44c4-9eef-18efb41051b8-kube-api-access-5gsjc" pod "ea5d6076-2ccf-44c4-9eef-18efb41051b8" (UID: "ea5d6076-2ccf-44c4-9eef-18efb41051b8"). InnerVolumeSpecName "kube-api-access-5gsjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:46.896512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.896433    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gsjc\" (UniqueName: \"kubernetes.io/projected/ea5d6076-2ccf-44c4-9eef-18efb41051b8-kube-api-access-5gsjc\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:46.896512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.896463    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:46.896512 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:46.896474    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5d6076-2ccf-44c4-9eef-18efb41051b8-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:47.294059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:47.294020    4447 generic.go:356] "Generic (PLEG): container finished" podID="ea5d6076-2ccf-44c4-9eef-18efb41051b8" containerID="72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d" exitCode=0
Nov 03 13:42:47.294576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:47.294071    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"
Nov 03 13:42:47.294576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:47.294109    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb" event={"ID":"ea5d6076-2ccf-44c4-9eef-18efb41051b8","Type":"ContainerDied","Data":"72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d"}
Nov 03 13:42:47.294576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:47.294186    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb" event={"ID":"ea5d6076-2ccf-44c4-9eef-18efb41051b8","Type":"ContainerDied","Data":"67f21a106b3b9c0fd58903da6af8903201264bc82650c3428ca8d944e01e2c0a"}
Nov 03 13:42:47.294576 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:47.294218    4447 scope.go:122] "RemoveContainer" containerID="72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d"
Nov 03 13:42:47.306642 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:47.306443    4447 scope.go:122] "RemoveContainer" containerID="72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d"
Nov 03 13:42:47.306805 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:42:47.306780    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d\": container with ID starting with 72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d not found: ID does not exist" containerID="72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d"
Nov 03 13:42:47.306853 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:47.306814    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d"} err="failed to get container status \"72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d\": rpc error: code = NotFound desc = could not find container \"72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d\": container with ID starting with 72dfb957b806039e87e6611943fbc39666fbf87d00863030bf3b92bbbcd94b0d not found: ID does not exist"
Nov 03 13:42:47.320062 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:47.320031    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"]
Nov 03 13:42:47.323185 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:47.323160    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb"]
Nov 03 13:42:48.319714 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:48.319680    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5d6076-2ccf-44c4-9eef-18efb41051b8" path="/var/lib/kubelet/pods/ea5d6076-2ccf-44c4-9eef-18efb41051b8/volumes"
Nov 03 13:42:49.353210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.353175    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"]
Nov 03 13:42:49.354059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.354033    4447 cpu_manager.go:414] "containerMap: removing container" podUID="ea5d6076-2ccf-44c4-9eef-18efb41051b8" containerName="maas-api"
Nov 03 13:42:49.354167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.354060    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5d6076-2ccf-44c4-9eef-18efb41051b8" containerName="maas-api"
Nov 03 13:42:49.354167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.354089    4447 cpu_manager.go:414] "containerMap: removing container" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy"
Nov 03 13:42:49.354167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.354096    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy"
Nov 03 13:42:49.354167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.354125    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy"
Nov 03 13:42:49.354167 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.354135    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy"
Nov 03 13:42:49.354331 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.354295    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5d6076-2ccf-44c4-9eef-18efb41051b8" containerName="maas-api"
Nov 03 13:42:49.354331 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.354309    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff4deac-8242-44bb-9b15-f7f57d2f7145" containerName="istio-proxy"
Nov 03 13:42:49.354331 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.354322    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73dcbee-73b7-4bd7-9254-09db37d787c9" containerName="istio-proxy"
Nov 03 13:42:49.359381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.359326    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.373210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.373186    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"]
Nov 03 13:42:49.427121 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.427083    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.427302 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.427256    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.427442 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.427312    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.427522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.427441    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.427522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.427463    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.427522 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.427482    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ns7x\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-kube-api-access-2ns7x\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.427672 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.427612    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.427672 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.427637    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.427774 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.427677    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529051    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529087 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529087    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529118    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ns7x\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-kube-api-access-2ns7x\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529182    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529200    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529237    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529410 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529279    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529682 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529425    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529682 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529476    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529682 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529568    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529682 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529649    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529770    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529829    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.529988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.529943    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.531599 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.531569    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.531803 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.531784    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.538008 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.537984    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.538008 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.537991    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ns7x\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-kube-api-access-2ns7x\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.619068 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.618976    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2b8a08-68fcfd885d-xblmb" podUID="ea5d6076-2ccf-44c4-9eef-18efb41051b8" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.124:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:42:49.676059 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.676016    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:49.979442 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.979403    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn" podUID="ca293290-1f26-4abf-a5ad-feccce1c7562" containerName="payload-processing" containerID="cri-o://0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01" gracePeriod=25
Nov 03 13:42:50.000794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:49.998616    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz" podUID="e7405dbd-4d03-4519-b204-bdb453592985" containerName="payload-pre-processing" containerID="cri-o://1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9" gracePeriod=25
Nov 03 13:42:50.057689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:50.057612    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"]
Nov 03 13:42:50.060172 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:42:50.059633    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1416aa_d93f_4ed6_a950_32adb7bcae7f.slice/crio-18589beba3f28aac797ff806c5e7b4b30cb616d036129874714bb1195d2f0008 WatchSource:0}: Error finding container 18589beba3f28aac797ff806c5e7b4b30cb616d036129874714bb1195d2f0008: Status 404 returned error can't find the container with id 18589beba3f28aac797ff806c5e7b4b30cb616d036129874714bb1195d2f0008
Nov 03 13:42:50.063177 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:50.063142    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:42:50.063403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:50.063228    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:42:50.063403 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:50.063273    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 13:42:50.321956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:50.321868    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" event={"ID":"ae1416aa-d93f-4ed6-a950-32adb7bcae7f","Type":"ContainerStarted","Data":"2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485"}
Nov 03 13:42:50.321956 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:50.321923    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" event={"ID":"ae1416aa-d93f-4ed6-a950-32adb7bcae7f","Type":"ContainerStarted","Data":"18589beba3f28aac797ff806c5e7b4b30cb616d036129874714bb1195d2f0008"}
Nov 03 13:42:50.346037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:50.345976    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" podStartSLOduration=1.345962167 podStartE2EDuration="1.345962167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:50.34133093 +0000 UTC m=+2058.152999148" watchObservedRunningTime="2026-11-03 13:42:50.345962167 +0000 UTC m=+2058.157630354"
Nov 03 13:42:50.676737 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:50.676633    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:51.678890 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:51.678851    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 13:42:51.679273 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:51.678907    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 13:42:52.677148 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:52.677114    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 13:42:52.677392 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:52.677169    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 13:42:52.721420 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:52.721383    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"]
Nov 03 13:42:52.721893 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:52.721595    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy" containerID="cri-o://b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc" gracePeriod=30
Nov 03 13:42:53.309883 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.309857    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:42:53.375199 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.375168    4447 generic.go:356] "Generic (PLEG): container finished" podID="47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" containerID="29c9a42920479721a116b273eac9bf5e995662db831f2532e6026922fe1efc5d" exitCode=0
Nov 03 13:42:53.375390 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.375264    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9" event={"ID":"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28","Type":"ContainerDied","Data":"29c9a42920479721a116b273eac9bf5e995662db831f2532e6026922fe1efc5d"}
Nov 03 13:42:53.376642 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.376617    4447 generic.go:356] "Generic (PLEG): container finished" podID="21a6efbf-968c-4e5d-93b3-a62eaae802d6" containerID="6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd" exitCode=0
Nov 03 13:42:53.376758 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.376667    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"
Nov 03 13:42:53.376758 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.376698    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz" event={"ID":"21a6efbf-968c-4e5d-93b3-a62eaae802d6","Type":"ContainerDied","Data":"6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd"}
Nov 03 13:42:53.376758 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.376743    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz" event={"ID":"21a6efbf-968c-4e5d-93b3-a62eaae802d6","Type":"ContainerDied","Data":"e3a02de32fe96504b813fc22f9ea87b8e4adc8b12990a40b92bf90a3125d956e"}
Nov 03 13:42:53.376913 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.376772    4447 scope.go:122] "RemoveContainer" containerID="6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd"
Nov 03 13:42:53.392627 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.392544    4447 scope.go:122] "RemoveContainer" containerID="6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd"
Nov 03 13:42:53.392955 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:42:53.392935    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd\": container with ID starting with 6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd not found: ID does not exist" containerID="6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd"
Nov 03 13:42:53.393046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.392967    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd"} err="failed to get container status \"6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd\": rpc error: code = NotFound desc = could not find container \"6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd\": container with ID starting with 6d597577a7dce9388e70c758131d156fd338c994c149f73e8593fa5e230445dd not found: ID does not exist"
Nov 03 13:42:53.393621 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.393599    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:42:53.403676 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.403656    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a6efbf-968c-4e5d-93b3-a62eaae802d6-kube-api-access-9bd8g\" (UniqueName: \"kubernetes.io/projected/21a6efbf-968c-4e5d-93b3-a62eaae802d6-kube-api-access-9bd8g\") pod \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\" (UID: \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\") "
Nov 03 13:42:53.403792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.403780    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a6efbf-968c-4e5d-93b3-a62eaae802d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6efbf-968c-4e5d-93b3-a62eaae802d6-plugins-config-volume\") pod \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\" (UID: \"21a6efbf-968c-4e5d-93b3-a62eaae802d6\") "
Nov 03 13:42:53.404120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.404099    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a6efbf-968c-4e5d-93b3-a62eaae802d6-plugins-config-volume" pod "21a6efbf-968c-4e5d-93b3-a62eaae802d6" (UID: "21a6efbf-968c-4e5d-93b3-a62eaae802d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:53.405680 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.405659    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a6efbf-968c-4e5d-93b3-a62eaae802d6-kube-api-access-9bd8g" pod "21a6efbf-968c-4e5d-93b3-a62eaae802d6" (UID: "21a6efbf-968c-4e5d-93b3-a62eaae802d6"). InnerVolumeSpecName "kube-api-access-9bd8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:53.505137 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.505102    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-kube-api-access-7llcj\" (UniqueName: \"kubernetes.io/projected/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-kube-api-access-7llcj\") pod \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\" (UID: \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\") "
Nov 03 13:42:53.505323 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.505282    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-plugins-config-volume\") pod \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\" (UID: \"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28\") "
Nov 03 13:42:53.505639 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.505619    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bd8g\" (UniqueName: \"kubernetes.io/projected/21a6efbf-968c-4e5d-93b3-a62eaae802d6-kube-api-access-9bd8g\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:53.505639 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.505640    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6efbf-968c-4e5d-93b3-a62eaae802d6-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:53.505784 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.505663    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-plugins-config-volume" pod "47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" (UID: "47cb9a4b-78ab-4a95-a88b-a9a2f83fac28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:53.507573 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.507546    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-kube-api-access-7llcj" pod "47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" (UID: "47cb9a4b-78ab-4a95-a88b-a9a2f83fac28"). InnerVolumeSpecName "kube-api-access-7llcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:53.606531 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.606444    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:53.606531 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.606474    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llcj\" (UniqueName: \"kubernetes.io/projected/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28-kube-api-access-7llcj\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:53.676834 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.676793    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 13:42:53.677017 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.676858    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 13:42:53.699145 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.699110    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"]
Nov 03 13:42:53.704976 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:53.704944    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a724a6-85d5f78fb7-ph7tz"]
Nov 03 13:42:54.320750 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:54.320673    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a6efbf-968c-4e5d-93b3-a62eaae802d6" path="/var/lib/kubelet/pods/21a6efbf-968c-4e5d-93b3-a62eaae802d6/volumes"
Nov 03 13:42:54.384803 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:54.384773    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"
Nov 03 13:42:54.384999 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:54.384776    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9" event={"ID":"47cb9a4b-78ab-4a95-a88b-a9a2f83fac28","Type":"ContainerDied","Data":"524e2c386a780e677c275eab928f0d2e52d0825f800e1bc6fac493c17c1f0c3b"}
Nov 03 13:42:54.384999 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:54.384928    4447 scope.go:122] "RemoveContainer" containerID="29c9a42920479721a116b273eac9bf5e995662db831f2532e6026922fe1efc5d"
Nov 03 13:42:54.401687 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:54.401654    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"]
Nov 03 13:42:54.413242 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:54.405590    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a724a6-74d7c8885f-ptql9"]
Nov 03 13:42:54.676608 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:54.676525    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 13:42:54.676608 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:54.676585    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 13:42:55.677232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:55.677196    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 13:42:55.677723 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:55.677264    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 13:42:56.320330 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:56.320296    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" path="/var/lib/kubelet/pods/47cb9a4b-78ab-4a95-a88b-a9a2f83fac28/volumes"
Nov 03 13:42:56.525393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:56.525320    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz" podUID="e7405dbd-4d03-4519-b204-bdb453592985" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:42:56.525393 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:56.525337    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn" podUID="ca293290-1f26-4abf-a5ad-feccce1c7562" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:42:56.676797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:56.676715    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 13:42:56.676797 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:56.676772    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 13:42:57.412013 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:57.411980    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:42:57.412435 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:57.412035    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:42:57.681380 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:57.681322    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:57.681652 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:57.681629    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:57.682316 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:57.682296    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:42:57.997782 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:57.997755    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:42:58.074957 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.074921    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-envoy\") pod \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") "
Nov 03 13:42:58.075124 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075015    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-podinfo\") pod \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") "
Nov 03 13:42:58.075124 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075075    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-certs\") pod \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") "
Nov 03 13:42:58.075231 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075191    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istiod-ca-cert\") pod \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") "
Nov 03 13:42:58.075284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075234    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-socket\") pod \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") "
Nov 03 13:42:58.075362 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075317    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-credential-socket\") pod \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") "
Nov 03 13:42:58.075428 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075369    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-token\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-token\") pod \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") "
Nov 03 13:42:58.075428 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075400    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-kube-api-access-28tvr\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-kube-api-access-28tvr\") pod \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") "
Nov 03 13:42:58.075527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075474    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-data\") pod \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\" (UID: \"da62ab1b-1934-4d0b-84be-a9e1627b65e7\") "
Nov 03 13:42:58.075527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075483    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-certs" pod "da62ab1b-1934-4d0b-84be-a9e1627b65e7" (UID: "da62ab1b-1934-4d0b-84be-a9e1627b65e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.075731 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075684    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istiod-ca-cert" pod "da62ab1b-1934-4d0b-84be-a9e1627b65e7" (UID: "da62ab1b-1934-4d0b-84be-a9e1627b65e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:58.075922 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075904    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.076003 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075925    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.076003 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.075901    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-data" pod "da62ab1b-1934-4d0b-84be-a9e1627b65e7" (UID: "da62ab1b-1934-4d0b-84be-a9e1627b65e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.076108 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.076029    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-socket" pod "da62ab1b-1934-4d0b-84be-a9e1627b65e7" (UID: "da62ab1b-1934-4d0b-84be-a9e1627b65e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.076108 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.076095    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-credential-socket" pod "da62ab1b-1934-4d0b-84be-a9e1627b65e7" (UID: "da62ab1b-1934-4d0b-84be-a9e1627b65e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.077333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.077307    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-podinfo" pod "da62ab1b-1934-4d0b-84be-a9e1627b65e7" (UID: "da62ab1b-1934-4d0b-84be-a9e1627b65e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:42:58.077333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.077319    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-envoy" pod "da62ab1b-1934-4d0b-84be-a9e1627b65e7" (UID: "da62ab1b-1934-4d0b-84be-a9e1627b65e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.077697 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.077615    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-token" pod "da62ab1b-1934-4d0b-84be-a9e1627b65e7" (UID: "da62ab1b-1934-4d0b-84be-a9e1627b65e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:58.077904 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.077884    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-kube-api-access-28tvr" pod "da62ab1b-1934-4d0b-84be-a9e1627b65e7" (UID: "da62ab1b-1934-4d0b-84be-a9e1627b65e7"). InnerVolumeSpecName "kube-api-access-28tvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:58.176792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.176760    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.176792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.176794    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.176792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.176809    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28tvr\" (UniqueName: \"kubernetes.io/projected/da62ab1b-1934-4d0b-84be-a9e1627b65e7-kube-api-access-28tvr\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.177066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.176825    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.177066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.176837    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.177066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.176849    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da62ab1b-1934-4d0b-84be-a9e1627b65e7-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.177066 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.176862    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da62ab1b-1934-4d0b-84be-a9e1627b65e7-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.415112 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.415070    4447 generic.go:356] "Generic (PLEG): container finished" podID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerID="b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc" exitCode=0
Nov 03 13:42:58.415624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.415125    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"
Nov 03 13:42:58.415624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.415159    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" event={"ID":"da62ab1b-1934-4d0b-84be-a9e1627b65e7","Type":"ContainerDied","Data":"b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc"}
Nov 03 13:42:58.415624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.415226    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779" event={"ID":"da62ab1b-1934-4d0b-84be-a9e1627b65e7","Type":"ContainerDied","Data":"f2ee005532dc9693db71492fccd1e847f4b35a0e9994eec5e93f7580a28f9d67"}
Nov 03 13:42:58.415624 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.415261    4447 scope.go:122] "RemoveContainer" containerID="b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc"
Nov 03 13:42:58.426095 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.426074    4447 scope.go:122] "RemoveContainer" containerID="b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc"
Nov 03 13:42:58.426446 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:42:58.426423    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc\": container with ID starting with b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc not found: ID does not exist" containerID="b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc"
Nov 03 13:42:58.426523 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.426454    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc"} err="failed to get container status \"b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc\": rpc error: code = NotFound desc = could not find container \"b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc\": container with ID starting with b120899ef98e8614e22383a312e870cac5057dd79afb78652ec2f65a61bbdcdc not found: ID does not exist"
Nov 03 13:42:58.432136 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.432114    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"]
Nov 03 13:42:58.441049 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:42:58.441019    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-h8779"]
Nov 03 13:43:00.319870 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:00.319839    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" path="/var/lib/kubelet/pods/da62ab1b-1934-4d0b-84be-a9e1627b65e7/volumes"
Nov 03 13:43:03.419178 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:43:03.419131    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:43:06.525305 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:06.525255    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn" podUID="ca293290-1f26-4abf-a5ad-feccce1c7562" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:06.525739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:06.525256    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz" podUID="e7405dbd-4d03-4519-b204-bdb453592985" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:10.296673 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.296648    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:43:10.368722 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.368698    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:43:10.443035 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.443001    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca293290-1f26-4abf-a5ad-feccce1c7562-kube-api-access-qf9hv\" (UniqueName: \"kubernetes.io/projected/ca293290-1f26-4abf-a5ad-feccce1c7562-kube-api-access-qf9hv\") pod \"ca293290-1f26-4abf-a5ad-feccce1c7562\" (UID: \"ca293290-1f26-4abf-a5ad-feccce1c7562\") "
Nov 03 13:43:10.443200 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.443151    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca293290-1f26-4abf-a5ad-feccce1c7562-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca293290-1f26-4abf-a5ad-feccce1c7562-plugins-config-volume\") pod \"ca293290-1f26-4abf-a5ad-feccce1c7562\" (UID: \"ca293290-1f26-4abf-a5ad-feccce1c7562\") "
Nov 03 13:43:10.443539 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.443516    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca293290-1f26-4abf-a5ad-feccce1c7562-plugins-config-volume" pod "ca293290-1f26-4abf-a5ad-feccce1c7562" (UID: "ca293290-1f26-4abf-a5ad-feccce1c7562"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:10.443789 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.443762    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca293290-1f26-4abf-a5ad-feccce1c7562-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:10.445237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.445216    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca293290-1f26-4abf-a5ad-feccce1c7562-kube-api-access-qf9hv" pod "ca293290-1f26-4abf-a5ad-feccce1c7562" (UID: "ca293290-1f26-4abf-a5ad-feccce1c7562"). InnerVolumeSpecName "kube-api-access-qf9hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:10.541843 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.541755    4447 generic.go:356] "Generic (PLEG): container finished" podID="e7405dbd-4d03-4519-b204-bdb453592985" containerID="1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9" exitCode=0
Nov 03 13:43:10.541843 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.541807    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"
Nov 03 13:43:10.542046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.541839    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz" event={"ID":"e7405dbd-4d03-4519-b204-bdb453592985","Type":"ContainerDied","Data":"1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9"}
Nov 03 13:43:10.542046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.541888    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz" event={"ID":"e7405dbd-4d03-4519-b204-bdb453592985","Type":"ContainerDied","Data":"681c096dd50946585cc9195eb43a87cb586cfdfcedec2eb612128c3683067c31"}
Nov 03 13:43:10.542046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.541917    4447 scope.go:122] "RemoveContainer" containerID="1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9"
Nov 03 13:43:10.543253 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.543232    4447 generic.go:356] "Generic (PLEG): container finished" podID="ca293290-1f26-4abf-a5ad-feccce1c7562" containerID="0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01" exitCode=0
Nov 03 13:43:10.543374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.543272    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"
Nov 03 13:43:10.543374 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.543325    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn" event={"ID":"ca293290-1f26-4abf-a5ad-feccce1c7562","Type":"ContainerDied","Data":"0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01"}
Nov 03 13:43:10.543469 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.543386    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn" event={"ID":"ca293290-1f26-4abf-a5ad-feccce1c7562","Type":"ContainerDied","Data":"70f746e800156ded84a7fb674c8559bad48d877ea326ef157f0eaa2f1a4983dc"}
Nov 03 13:43:10.544139 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.544118    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7405dbd-4d03-4519-b204-bdb453592985-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7405dbd-4d03-4519-b204-bdb453592985-plugins-config-volume\") pod \"e7405dbd-4d03-4519-b204-bdb453592985\" (UID: \"e7405dbd-4d03-4519-b204-bdb453592985\") "
Nov 03 13:43:10.544213 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.544199    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7405dbd-4d03-4519-b204-bdb453592985-kube-api-access-zq46f\" (UniqueName: \"kubernetes.io/projected/e7405dbd-4d03-4519-b204-bdb453592985-kube-api-access-zq46f\") pod \"e7405dbd-4d03-4519-b204-bdb453592985\" (UID: \"e7405dbd-4d03-4519-b204-bdb453592985\") "
Nov 03 13:43:10.544599 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.544570    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7405dbd-4d03-4519-b204-bdb453592985-plugins-config-volume" pod "e7405dbd-4d03-4519-b204-bdb453592985" (UID: "e7405dbd-4d03-4519-b204-bdb453592985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:10.544788 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.544769    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf9hv\" (UniqueName: \"kubernetes.io/projected/ca293290-1f26-4abf-a5ad-feccce1c7562-kube-api-access-qf9hv\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:10.544869 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.544788    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7405dbd-4d03-4519-b204-bdb453592985-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:10.546391 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.546363    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7405dbd-4d03-4519-b204-bdb453592985-kube-api-access-zq46f" pod "e7405dbd-4d03-4519-b204-bdb453592985" (UID: "e7405dbd-4d03-4519-b204-bdb453592985"). InnerVolumeSpecName "kube-api-access-zq46f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:10.552288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.552264    4447 scope.go:122] "RemoveContainer" containerID="1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9"
Nov 03 13:43:10.552620 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:43:10.552598    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9\": container with ID starting with 1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9 not found: ID does not exist" containerID="1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9"
Nov 03 13:43:10.552727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.552625    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9"} err="failed to get container status \"1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9\": rpc error: code = NotFound desc = could not find container \"1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9\": container with ID starting with 1a27e2cbbe73942f00fd69087939e2354885083b4b3a73197a6290b4407364a9 not found: ID does not exist"
Nov 03 13:43:10.552727 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.552643    4447 scope.go:122] "RemoveContainer" containerID="0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01"
Nov 03 13:43:10.561615 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.561600    4447 scope.go:122] "RemoveContainer" containerID="0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01"
Nov 03 13:43:10.561904 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:43:10.561885    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01\": container with ID starting with 0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01 not found: ID does not exist" containerID="0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01"
Nov 03 13:43:10.561963 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.561910    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01"} err="failed to get container status \"0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01\": rpc error: code = NotFound desc = could not find container \"0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01\": container with ID starting with 0c2b0a77e04d0d63edf8b9707c3d654cebf211781136ba476338a4a734354c01 not found: ID does not exist"
Nov 03 13:43:10.568129 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.568108    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"]
Nov 03 13:43:10.572617 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.572597    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2b8a08-d76f44b9-6xbbn"]
Nov 03 13:43:10.646106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.646072    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq46f\" (UniqueName: \"kubernetes.io/projected/e7405dbd-4d03-4519-b204-bdb453592985-kube-api-access-zq46f\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:10.867046 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.867012    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"]
Nov 03 13:43:10.872874 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:10.872844    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2b8a08-7c9fc5b559-mslxz"]
Nov 03 13:43:12.329364 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:12.329313    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca293290-1f26-4abf-a5ad-feccce1c7562" path="/var/lib/kubelet/pods/ca293290-1f26-4abf-a5ad-feccce1c7562/volumes"
Nov 03 13:43:12.329950 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:12.329929    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7405dbd-4d03-4519-b204-bdb453592985" path="/var/lib/kubelet/pods/e7405dbd-4d03-4519-b204-bdb453592985/volumes"
Nov 03 13:43:24.809677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:24.809642    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"]
Nov 03 13:43:24.810134 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:24.809874    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy" containerID="cri-o://db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9" gracePeriod=30
Nov 03 13:43:30.083001 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.082930    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:43:30.204389 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204355    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-credential-socket\") pod \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") "
Nov 03 13:43:30.204554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204416    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-token\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-token\") pod \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") "
Nov 03 13:43:30.204554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204466    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-data\") pod \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") "
Nov 03 13:43:30.204554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204510    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istiod-ca-cert\") pod \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") "
Nov 03 13:43:30.204554 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204550    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-kube-api-access-gfkmw\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-kube-api-access-gfkmw\") pod \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") "
Nov 03 13:43:30.204776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204634    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-certs\") pod \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") "
Nov 03 13:43:30.204776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204651    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-credential-socket" pod "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" (UID: "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:30.204776 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204667    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-podinfo\") pod \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") "
Nov 03 13:43:30.204946 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204780    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-socket\") pod \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") "
Nov 03 13:43:30.204946 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204787    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-data" pod "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" (UID: "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:30.204946 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.204827    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-envoy\") pod \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\" (UID: \"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5\") "
Nov 03 13:43:30.205119 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.205049    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-certs" pod "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" (UID: "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:30.205119 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.205066    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-socket" pod "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" (UID: "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:30.205431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.205397    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istiod-ca-cert" pod "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" (UID: "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:30.205431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.205407    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:30.205431 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.205429    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:30.205608 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.205443    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:30.205608 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.205455    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:30.206855 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.206831    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-kube-api-access-gfkmw" pod "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" (UID: "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5"). InnerVolumeSpecName "kube-api-access-gfkmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:30.207476 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.207446    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-token" pod "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" (UID: "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:30.207571 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.207463    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-podinfo" pod "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" (UID: "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:43:30.207571 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.207474    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-envoy" pod "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" (UID: "f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:30.307286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.307247    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:30.307286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.307283    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:30.307286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.307298    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfkmw\" (UniqueName: \"kubernetes.io/projected/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-kube-api-access-gfkmw\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:30.307647 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.307307    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:30.307647 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.307315    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:30.684283 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.684243    4447 generic.go:356] "Generic (PLEG): container finished" podID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerID="db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9" exitCode=0
Nov 03 13:43:30.684495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.684296    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"
Nov 03 13:43:30.684495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.684466    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" event={"ID":"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5","Type":"ContainerDied","Data":"db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9"}
Nov 03 13:43:30.684609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.684541    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv" event={"ID":"f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5","Type":"ContainerDied","Data":"a4c3962203798698257327a847034f80f4c4821317445bb7253c88646b66f32b"}
Nov 03 13:43:30.684609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.684582    4447 scope.go:122] "RemoveContainer" containerID="db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9"
Nov 03 13:43:30.698798 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.698544    4447 scope.go:122] "RemoveContainer" containerID="db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9"
Nov 03 13:43:30.702432 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:43:30.702405    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9\": container with ID starting with db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9 not found: ID does not exist" containerID="db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9"
Nov 03 13:43:30.702547 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.702443    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9"} err="failed to get container status \"db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9\": rpc error: code = NotFound desc = could not find container \"db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9\": container with ID starting with db3ea157868367d308d498b3a6e48c107814500fc7299a80fbca27424db46ba9 not found: ID does not exist"
Nov 03 13:43:30.708177 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.708150    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"]
Nov 03 13:43:30.727040 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:30.727001    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-v2xtv"]
Nov 03 13:43:32.321200 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:43:32.321168    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" path="/var/lib/kubelet/pods/f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5/volumes"
Nov 03 13:43:33.412702 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:43:33.412665    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:44:00.378816 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.378735    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"]
Nov 03 13:44:00.379329 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.378983    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx" podUID="da579797-607a-4132-945d-69c919276f0d" containerName="maas-api" containerID="cri-o://b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f" gracePeriod=30
Nov 03 13:44:00.740691 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.740515    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:44:00.879909 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.879874    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-metrics-certs\") pod \"da579797-607a-4132-945d-69c919276f0d\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") "
Nov 03 13:44:00.880120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.879922    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da579797-607a-4132-945d-69c919276f0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da579797-607a-4132-945d-69c919276f0d-openshift-service-ca\") pod \"da579797-607a-4132-945d-69c919276f0d\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") "
Nov 03 13:44:00.880120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.879952    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da579797-607a-4132-945d-69c919276f0d-kube-api-access-tf8sn\" (UniqueName: \"kubernetes.io/projected/da579797-607a-4132-945d-69c919276f0d-kube-api-access-tf8sn\") pod \"da579797-607a-4132-945d-69c919276f0d\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") "
Nov 03 13:44:00.880120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.879989    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls\") pod \"da579797-607a-4132-945d-69c919276f0d\" (UID: \"da579797-607a-4132-945d-69c919276f0d\") "
Nov 03 13:44:00.880445 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.880419    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da579797-607a-4132-945d-69c919276f0d-openshift-service-ca" pod "da579797-607a-4132-945d-69c919276f0d" (UID: "da579797-607a-4132-945d-69c919276f0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:00.882650 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.882612    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-metrics-certs" pod "da579797-607a-4132-945d-69c919276f0d" (UID: "da579797-607a-4132-945d-69c919276f0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:00.882789 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.882732    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls" pod "da579797-607a-4132-945d-69c919276f0d" (UID: "da579797-607a-4132-945d-69c919276f0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:00.882789 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.882768    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da579797-607a-4132-945d-69c919276f0d-kube-api-access-tf8sn" pod "da579797-607a-4132-945d-69c919276f0d" (UID: "da579797-607a-4132-945d-69c919276f0d"). InnerVolumeSpecName "kube-api-access-tf8sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:00.924900 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.924861    4447 generic.go:356] "Generic (PLEG): container finished" podID="da579797-607a-4132-945d-69c919276f0d" containerID="b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f" exitCode=0
Nov 03 13:44:00.925633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.924943    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"
Nov 03 13:44:00.925633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.924956    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx" event={"ID":"da579797-607a-4132-945d-69c919276f0d","Type":"ContainerDied","Data":"b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f"}
Nov 03 13:44:00.925633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.925018    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx" event={"ID":"da579797-607a-4132-945d-69c919276f0d","Type":"ContainerDied","Data":"d33b181259245f935100407c29f3a79113270b836ac6e5060708ac14e50ffd8a"}
Nov 03 13:44:00.925633 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.925061    4447 scope.go:122] "RemoveContainer" containerID="b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f"
Nov 03 13:44:00.940099 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.940079    4447 scope.go:122] "RemoveContainer" containerID="b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f"
Nov 03 13:44:00.940516 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:44:00.940493    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f\": container with ID starting with b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f not found: ID does not exist" containerID="b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f"
Nov 03 13:44:00.940629 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.940526    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f"} err="failed to get container status \"b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f\": rpc error: code = NotFound desc = could not find container \"b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f\": container with ID starting with b1906e10ff24ad5e14c7a17cd462ea67f2a705f585987dbce2c0905d9e162c8f not found: ID does not exist"
Nov 03 13:44:00.952404 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.952297    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"]
Nov 03 13:44:00.958960 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.958940    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c47670-54846957c5-mq2cx"]
Nov 03 13:44:00.981675 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.981647    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:00.981675 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.981673    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da579797-607a-4132-945d-69c919276f0d-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:00.981828 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.981685    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8sn\" (UniqueName: \"kubernetes.io/projected/da579797-607a-4132-945d-69c919276f0d-kube-api-access-tf8sn\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:00.981828 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:00.981694    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da579797-607a-4132-945d-69c919276f0d-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:02.319661 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:02.319629    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da579797-607a-4132-945d-69c919276f0d" path="/var/lib/kubelet/pods/da579797-607a-4132-945d-69c919276f0d/volumes"
Nov 03 13:44:03.411406 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:44:03.411368    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:44:04.900010 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:04.899975    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"]
Nov 03 13:44:04.900467 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:04.900196    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy" containerID="cri-o://2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272" gracePeriod=30
Nov 03 13:44:04.921586 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:04.921547    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"]
Nov 03 13:44:04.921814 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:04.921777    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy" containerID="cri-o://8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab" gracePeriod=30
Nov 03 13:44:04.930226 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:04.930195    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"]
Nov 03 13:44:04.930442 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:04.930420    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy" containerID="cri-o://2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485" gracePeriod=30
Nov 03 13:44:08.794644 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:08.794595    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:44:08.795140 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:08.794671    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:44:10.199253 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.199221    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:44:10.292489 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.292462    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:44:10.295698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.295673    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:44:10.302859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.302837    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-envoy\") pod \"dde47a9b-1631-445e-bd1f-98e1da600c0a\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") "
Nov 03 13:44:10.302997 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.302882    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-certs\") pod \"dde47a9b-1631-445e-bd1f-98e1da600c0a\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") "
Nov 03 13:44:10.302997 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.302939    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-credential-socket\") pod \"dde47a9b-1631-445e-bd1f-98e1da600c0a\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") "
Nov 03 13:44:10.303180 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303033    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-token\") pod \"dde47a9b-1631-445e-bd1f-98e1da600c0a\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") "
Nov 03 13:44:10.303180 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303094    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-kube-api-access-j5ngl\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-kube-api-access-j5ngl\") pod \"dde47a9b-1631-445e-bd1f-98e1da600c0a\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") "
Nov 03 13:44:10.303298 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303186    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-socket\") pod \"dde47a9b-1631-445e-bd1f-98e1da600c0a\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") "
Nov 03 13:44:10.303298 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303217    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-certs" pod "dde47a9b-1631-445e-bd1f-98e1da600c0a" (UID: "dde47a9b-1631-445e-bd1f-98e1da600c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.303298 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303223    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-data\") pod \"dde47a9b-1631-445e-bd1f-98e1da600c0a\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") "
Nov 03 13:44:10.303495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303306    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde47a9b-1631-445e-bd1f-98e1da600c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde47a9b-1631-445e-bd1f-98e1da600c0a-istiod-ca-cert\") pod \"dde47a9b-1631-445e-bd1f-98e1da600c0a\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") "
Nov 03 13:44:10.303495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303325    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-credential-socket" pod "dde47a9b-1631-445e-bd1f-98e1da600c0a" (UID: "dde47a9b-1631-445e-bd1f-98e1da600c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.303495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303387    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-podinfo\") pod \"dde47a9b-1631-445e-bd1f-98e1da600c0a\" (UID: \"dde47a9b-1631-445e-bd1f-98e1da600c0a\") "
Nov 03 13:44:10.303495 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303441    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-data" pod "dde47a9b-1631-445e-bd1f-98e1da600c0a" (UID: "dde47a9b-1631-445e-bd1f-98e1da600c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.303712 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303548    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-socket" pod "dde47a9b-1631-445e-bd1f-98e1da600c0a" (UID: "dde47a9b-1631-445e-bd1f-98e1da600c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.303712 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303671    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde47a9b-1631-445e-bd1f-98e1da600c0a-istiod-ca-cert" pod "dde47a9b-1631-445e-bd1f-98e1da600c0a" (UID: "dde47a9b-1631-445e-bd1f-98e1da600c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:10.303974 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.303951    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.304111 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.304096    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.304223 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.304208    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde47a9b-1631-445e-bd1f-98e1da600c0a-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.304332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.304317    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.304490 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.304473    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.305511 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.305486    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-kube-api-access-j5ngl" pod "dde47a9b-1631-445e-bd1f-98e1da600c0a" (UID: "dde47a9b-1631-445e-bd1f-98e1da600c0a"). InnerVolumeSpecName "kube-api-access-j5ngl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.305751 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.305730    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-envoy" pod "dde47a9b-1631-445e-bd1f-98e1da600c0a" (UID: "dde47a9b-1631-445e-bd1f-98e1da600c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.305899 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.305885    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-podinfo" pod "dde47a9b-1631-445e-bd1f-98e1da600c0a" (UID: "dde47a9b-1631-445e-bd1f-98e1da600c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:10.305979 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.305967    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-token" pod "dde47a9b-1631-445e-bd1f-98e1da600c0a" (UID: "dde47a9b-1631-445e-bd1f-98e1da600c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.405152 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.405110    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-socket\") pod \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") "
Nov 03 13:44:10.405152 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.405200    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-credential-socket\") pod \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") "
Nov 03 13:44:10.405152 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.405241    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-certs\") pod \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") "
Nov 03 13:44:10.405152 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.405463    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-socket" pod "d6b45341-138a-4e2e-a5bd-85fc060d7e17" (UID: "d6b45341-138a-4e2e-a5bd-85fc060d7e17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.405865 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.405689    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-credential-socket" pod "ae1416aa-d93f-4ed6-a950-32adb7bcae7f" (UID: "ae1416aa-d93f-4ed6-a950-32adb7bcae7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.405926 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.405901    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-certs" pod "ae1416aa-d93f-4ed6-a950-32adb7bcae7f" (UID: "ae1416aa-d93f-4ed6-a950-32adb7bcae7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.406021 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.405284    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-envoy\") pod \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") "
Nov 03 13:44:10.406128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406063    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-podinfo\") pod \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") "
Nov 03 13:44:10.406128 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406102    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-certs\") pod \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") "
Nov 03 13:44:10.406232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406135    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istiod-ca-cert\") pod \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") "
Nov 03 13:44:10.406232 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406205    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-token\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-token\") pod \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") "
Nov 03 13:44:10.406332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406305    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istiod-ca-cert\") pod \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") "
Nov 03 13:44:10.406437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406363    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-podinfo\") pod \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") "
Nov 03 13:44:10.406437 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406404    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-credential-socket\") pod \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") "
Nov 03 13:44:10.406545 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406449    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-token\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-token\") pod \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") "
Nov 03 13:44:10.406545 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406493    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-data\") pod \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") "
Nov 03 13:44:10.406545 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406526    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-data\") pod \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") "
Nov 03 13:44:10.406698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406563    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-kube-api-access-2ns7x\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-kube-api-access-2ns7x\") pod \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") "
Nov 03 13:44:10.406698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406596    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-envoy\") pod \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") "
Nov 03 13:44:10.406698 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406679    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-socket\") pod \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\" (UID: \"ae1416aa-d93f-4ed6-a950-32adb7bcae7f\") "
Nov 03 13:44:10.406836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.406713    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-kube-api-access-xzj5z\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-kube-api-access-xzj5z\") pod \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\" (UID: \"d6b45341-138a-4e2e-a5bd-85fc060d7e17\") "
Nov 03 13:44:10.407369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.407220    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.407369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.407245    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.407369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.407259    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.407369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.407274    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.407369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.407289    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.407369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.407303    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.407369 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.407317    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5ngl\" (UniqueName: \"kubernetes.io/projected/dde47a9b-1631-445e-bd1f-98e1da600c0a-kube-api-access-j5ngl\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.407822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.407519    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-credential-socket" pod "d6b45341-138a-4e2e-a5bd-85fc060d7e17" (UID: "d6b45341-138a-4e2e-a5bd-85fc060d7e17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.407822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.407570    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istiod-ca-cert" pod "ae1416aa-d93f-4ed6-a950-32adb7bcae7f" (UID: "ae1416aa-d93f-4ed6-a950-32adb7bcae7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:10.409189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.408309    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istiod-ca-cert" pod "d6b45341-138a-4e2e-a5bd-85fc060d7e17" (UID: "d6b45341-138a-4e2e-a5bd-85fc060d7e17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:10.409189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.408587    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-certs" pod "d6b45341-138a-4e2e-a5bd-85fc060d7e17" (UID: "d6b45341-138a-4e2e-a5bd-85fc060d7e17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.409189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.408754    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-data" pod "ae1416aa-d93f-4ed6-a950-32adb7bcae7f" (UID: "ae1416aa-d93f-4ed6-a950-32adb7bcae7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.409189 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.408934    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-socket" pod "ae1416aa-d93f-4ed6-a950-32adb7bcae7f" (UID: "ae1416aa-d93f-4ed6-a950-32adb7bcae7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.409759 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.409730    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-data" pod "d6b45341-138a-4e2e-a5bd-85fc060d7e17" (UID: "d6b45341-138a-4e2e-a5bd-85fc060d7e17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.414328 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.414305    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-envoy" pod "ae1416aa-d93f-4ed6-a950-32adb7bcae7f" (UID: "ae1416aa-d93f-4ed6-a950-32adb7bcae7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.415866 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.415758    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-envoy" pod "d6b45341-138a-4e2e-a5bd-85fc060d7e17" (UID: "d6b45341-138a-4e2e-a5bd-85fc060d7e17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.415866 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.415782    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-podinfo" pod "d6b45341-138a-4e2e-a5bd-85fc060d7e17" (UID: "d6b45341-138a-4e2e-a5bd-85fc060d7e17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:10.415866 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.415790    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-kube-api-access-2ns7x" pod "ae1416aa-d93f-4ed6-a950-32adb7bcae7f" (UID: "ae1416aa-d93f-4ed6-a950-32adb7bcae7f"). InnerVolumeSpecName "kube-api-access-2ns7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.415866 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.415849    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-token" pod "ae1416aa-d93f-4ed6-a950-32adb7bcae7f" (UID: "ae1416aa-d93f-4ed6-a950-32adb7bcae7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.416089 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.415890    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-token" pod "d6b45341-138a-4e2e-a5bd-85fc060d7e17" (UID: "d6b45341-138a-4e2e-a5bd-85fc060d7e17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.416089 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.415918    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-podinfo" pod "ae1416aa-d93f-4ed6-a950-32adb7bcae7f" (UID: "ae1416aa-d93f-4ed6-a950-32adb7bcae7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:10.417123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.417102    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-kube-api-access-xzj5z" pod "d6b45341-138a-4e2e-a5bd-85fc060d7e17" (UID: "d6b45341-138a-4e2e-a5bd-85fc060d7e17"). InnerVolumeSpecName "kube-api-access-xzj5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.508171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508122    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508160    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508170    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508171 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508179    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508188    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508196    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508205    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ns7x\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-kube-api-access-2ns7x\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508216    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508224    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508232    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzj5z\" (UniqueName: \"kubernetes.io/projected/d6b45341-138a-4e2e-a5bd-85fc060d7e17-kube-api-access-xzj5z\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508240    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508248    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508255    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b45341-138a-4e2e-a5bd-85fc060d7e17-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508263    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.508567 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:10.508271    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1416aa-d93f-4ed6-a950-32adb7bcae7f-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:11.018463 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.018423    4447 generic.go:356] "Generic (PLEG): container finished" podID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerID="2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485" exitCode=0
Nov 03 13:44:11.018668 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.018475    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"
Nov 03 13:44:11.018668 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.018518    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" event={"ID":"ae1416aa-d93f-4ed6-a950-32adb7bcae7f","Type":"ContainerDied","Data":"2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485"}
Nov 03 13:44:11.018668 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.018587    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c" event={"ID":"ae1416aa-d93f-4ed6-a950-32adb7bcae7f","Type":"ContainerDied","Data":"18589beba3f28aac797ff806c5e7b4b30cb616d036129874714bb1195d2f0008"}
Nov 03 13:44:11.018668 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.018617    4447 scope.go:122] "RemoveContainer" containerID="2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485"
Nov 03 13:44:11.020097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.020062    4447 generic.go:356] "Generic (PLEG): container finished" podID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerID="8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab" exitCode=0
Nov 03 13:44:11.020097 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.020082    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" event={"ID":"d6b45341-138a-4e2e-a5bd-85fc060d7e17","Type":"ContainerDied","Data":"8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab"}
Nov 03 13:44:11.020268 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.020115    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"
Nov 03 13:44:11.020268 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.020132    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5" event={"ID":"d6b45341-138a-4e2e-a5bd-85fc060d7e17","Type":"ContainerDied","Data":"2bc22f08f7cb162b2b98e6a757d1bec3a9b94ad7c2516214a3602df46dc70bfe"}
Nov 03 13:44:11.021821 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.021798    4447 generic.go:356] "Generic (PLEG): container finished" podID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerID="2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272" exitCode=0
Nov 03 13:44:11.021929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.021876    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" event={"ID":"dde47a9b-1631-445e-bd1f-98e1da600c0a","Type":"ContainerDied","Data":"2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272"}
Nov 03 13:44:11.021929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.021915    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4" event={"ID":"dde47a9b-1631-445e-bd1f-98e1da600c0a","Type":"ContainerDied","Data":"ee676bb9996fc4294c1caa4fba60fe295bfd4452cddd7042b02a99147cf07e4b"}
Nov 03 13:44:11.022045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.021993    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"
Nov 03 13:44:11.031262 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.031223    4447 scope.go:122] "RemoveContainer" containerID="2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485"
Nov 03 13:44:11.031690 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:44:11.031573    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485\": container with ID starting with 2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485 not found: ID does not exist" containerID="2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485"
Nov 03 13:44:11.031690 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.031607    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485"} err="failed to get container status \"2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485\": rpc error: code = NotFound desc = could not find container \"2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485\": container with ID starting with 2d27ea2b6b147d3587dfb911b0bbeb0a0728494f96b0d4dcf5a0141c1cf43485 not found: ID does not exist"
Nov 03 13:44:11.031690 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.031631    4447 scope.go:122] "RemoveContainer" containerID="8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab"
Nov 03 13:44:11.041772 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.041752    4447 scope.go:122] "RemoveContainer" containerID="8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab"
Nov 03 13:44:11.042052 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:44:11.042032    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab\": container with ID starting with 8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab not found: ID does not exist" containerID="8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab"
Nov 03 13:44:11.042119 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.042059    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab"} err="failed to get container status \"8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab\": rpc error: code = NotFound desc = could not find container \"8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab\": container with ID starting with 8750e6dc3fd3ea3a9066f461eace63ca22d36e8ca1e6b977f4faddea9b3fbeab not found: ID does not exist"
Nov 03 13:44:11.042119 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.042075    4447 scope.go:122] "RemoveContainer" containerID="2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272"
Nov 03 13:44:11.051659 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.051638    4447 scope.go:122] "RemoveContainer" containerID="2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272"
Nov 03 13:44:11.051990 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:44:11.051966    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272\": container with ID starting with 2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272 not found: ID does not exist" containerID="2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272"
Nov 03 13:44:11.052063 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.052001    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272"} err="failed to get container status \"2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272\": rpc error: code = NotFound desc = could not find container \"2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272\": container with ID starting with 2faf8d2dbbbbdc1ba8c979a8c90580bdef26c43e294d90b6b4b638f4d710c272 not found: ID does not exist"
Nov 03 13:44:11.052307 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.052264    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"]
Nov 03 13:44:11.056061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.056039    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-hq2g4"]
Nov 03 13:44:11.070411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.070383    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"]
Nov 03 13:44:11.075284 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.075261    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-h5l5c"]
Nov 03 13:44:11.093634 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.093603    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"]
Nov 03 13:44:11.100077 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:11.100055    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-2bth5"]
Nov 03 13:44:12.322448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:12.322410    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" path="/var/lib/kubelet/pods/ae1416aa-d93f-4ed6-a950-32adb7bcae7f/volumes"
Nov 03 13:44:12.322947 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:12.322868    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" path="/var/lib/kubelet/pods/d6b45341-138a-4e2e-a5bd-85fc060d7e17/volumes"
Nov 03 13:44:12.323270 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:12.323253    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" path="/var/lib/kubelet/pods/dde47a9b-1631-445e-bd1f-98e1da600c0a/volumes"
Nov 03 13:44:18.465446 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.465408    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"]
Nov 03 13:44:18.466106 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.465711    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9" podUID="4b570f62-46d4-4b89-a7e1-9dc04c728f66" containerName="maas-api" containerID="cri-o://95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3" gracePeriod=30
Nov 03 13:44:18.777262 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.777236    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:44:18.917882 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.917852    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-metrics-certs\") pod \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") "
Nov 03 13:44:18.918084 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.917889    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b570f62-46d4-4b89-a7e1-9dc04c728f66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b570f62-46d4-4b89-a7e1-9dc04c728f66-openshift-service-ca\") pod \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") "
Nov 03 13:44:18.918084 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.917927    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-maas-api-tls\") pod \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") "
Nov 03 13:44:18.918211 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.918097    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b570f62-46d4-4b89-a7e1-9dc04c728f66-kube-api-access-44bnc\" (UniqueName: \"kubernetes.io/projected/4b570f62-46d4-4b89-a7e1-9dc04c728f66-kube-api-access-44bnc\") pod \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\" (UID: \"4b570f62-46d4-4b89-a7e1-9dc04c728f66\") "
Nov 03 13:44:18.918332 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.918302    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b570f62-46d4-4b89-a7e1-9dc04c728f66-openshift-service-ca" pod "4b570f62-46d4-4b89-a7e1-9dc04c728f66" (UID: "4b570f62-46d4-4b89-a7e1-9dc04c728f66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:18.920453 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.918569    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b570f62-46d4-4b89-a7e1-9dc04c728f66-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:18.920593 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.920558    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-metrics-certs" pod "4b570f62-46d4-4b89-a7e1-9dc04c728f66" (UID: "4b570f62-46d4-4b89-a7e1-9dc04c728f66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:18.920662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.920593    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b570f62-46d4-4b89-a7e1-9dc04c728f66-kube-api-access-44bnc" pod "4b570f62-46d4-4b89-a7e1-9dc04c728f66" (UID: "4b570f62-46d4-4b89-a7e1-9dc04c728f66"). InnerVolumeSpecName "kube-api-access-44bnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:18.920768 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:18.920728    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-maas-api-tls" pod "4b570f62-46d4-4b89-a7e1-9dc04c728f66" (UID: "4b570f62-46d4-4b89-a7e1-9dc04c728f66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:19.019411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.019301    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:19.019411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.019334    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b570f62-46d4-4b89-a7e1-9dc04c728f66-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:19.019411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.019374    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44bnc\" (UniqueName: \"kubernetes.io/projected/4b570f62-46d4-4b89-a7e1-9dc04c728f66-kube-api-access-44bnc\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:19.076602 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.076563    4447 generic.go:356] "Generic (PLEG): container finished" podID="4b570f62-46d4-4b89-a7e1-9dc04c728f66" containerID="95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3" exitCode=0
Nov 03 13:44:19.076767 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.076612    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"
Nov 03 13:44:19.076767 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.076643    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9" event={"ID":"4b570f62-46d4-4b89-a7e1-9dc04c728f66","Type":"ContainerDied","Data":"95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3"}
Nov 03 13:44:19.076767 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.076699    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9" event={"ID":"4b570f62-46d4-4b89-a7e1-9dc04c728f66","Type":"ContainerDied","Data":"6f8e19df83814db10d12e0a7773be94ed26c69faa18e5bb045c74a3a9c3cdc5c"}
Nov 03 13:44:19.076767 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.076727    4447 scope.go:122] "RemoveContainer" containerID="95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3"
Nov 03 13:44:19.087036 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.087017    4447 scope.go:122] "RemoveContainer" containerID="95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3"
Nov 03 13:44:19.087301 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:44:19.087279    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3\": container with ID starting with 95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3 not found: ID does not exist" containerID="95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3"
Nov 03 13:44:19.087404 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.087315    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3"} err="failed to get container status \"95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3\": rpc error: code = NotFound desc = could not find container \"95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3\": container with ID starting with 95849efa807a1f5502fb20c84b67965ff1eff38b3f9ff8ca4f4e1fd82d7219a3 not found: ID does not exist"
Nov 03 13:44:19.099780 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.099747    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"]
Nov 03 13:44:19.103762 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:19.103740    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9f71f2-64b796f598-lctm9"]
Nov 03 13:44:20.319829 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:20.319791    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b570f62-46d4-4b89-a7e1-9dc04c728f66" path="/var/lib/kubelet/pods/4b570f62-46d4-4b89-a7e1-9dc04c728f66/volumes"
Nov 03 13:44:33.412805 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:44:33.412762    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:44:37.170905 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.170854    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"]
Nov 03 13:44:37.171429 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.171241    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm" podUID="61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" containerName="maas-api" containerID="cri-o://a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea" gracePeriod=30
Nov 03 13:44:37.481197 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.481162    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:44:37.553807 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.553777    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-kube-api-access-4g9mn\" (UniqueName: \"kubernetes.io/projected/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-kube-api-access-4g9mn\") pod \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") "
Nov 03 13:44:37.553985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.553835    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-metrics-certs\") pod \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") "
Nov 03 13:44:37.553985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.553881    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-openshift-service-ca\") pod \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") "
Nov 03 13:44:37.553985 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.553961    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-maas-api-tls\") pod \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\" (UID: \"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8\") "
Nov 03 13:44:37.554325 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.554299    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-openshift-service-ca" pod "61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" (UID: "61c4cac3-b6f0-4f9f-b6fd-20105519b6c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:37.556388 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.556357    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-maas-api-tls" pod "61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" (UID: "61c4cac3-b6f0-4f9f-b6fd-20105519b6c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:37.556388 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.556383    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-kube-api-access-4g9mn" pod "61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" (UID: "61c4cac3-b6f0-4f9f-b6fd-20105519b6c8"). InnerVolumeSpecName "kube-api-access-4g9mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:37.556388 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.556418    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-metrics-certs" pod "61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" (UID: "61c4cac3-b6f0-4f9f-b6fd-20105519b6c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:37.654972 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.654942    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:37.654972 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.654969    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g9mn\" (UniqueName: \"kubernetes.io/projected/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-kube-api-access-4g9mn\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:37.655178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.654986    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:37.655178 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:37.654994    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:38.201310 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.201273    4447 generic.go:356] "Generic (PLEG): container finished" podID="61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" containerID="a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea" exitCode=0
Nov 03 13:44:38.201836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.201323    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"
Nov 03 13:44:38.201836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.201339    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm" event={"ID":"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8","Type":"ContainerDied","Data":"a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea"}
Nov 03 13:44:38.201836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.201409    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm" event={"ID":"61c4cac3-b6f0-4f9f-b6fd-20105519b6c8","Type":"ContainerDied","Data":"a67717bfc8c44e6981b081148a7e3bc561f8ea7c7496c7ad41beb5792bc5d15e"}
Nov 03 13:44:38.201836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.201440    4447 scope.go:122] "RemoveContainer" containerID="a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea"
Nov 03 13:44:38.211780 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.211756    4447 scope.go:122] "RemoveContainer" containerID="a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea"
Nov 03 13:44:38.212044 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:44:38.212027    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea\": container with ID starting with a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea not found: ID does not exist" containerID="a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea"
Nov 03 13:44:38.212107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.212058    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea"} err="failed to get container status \"a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea\": rpc error: code = NotFound desc = could not find container \"a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea\": container with ID starting with a8d3dcb85d2035ec707fde59e113df2a15fa97d9d7bac566a1562571651586ea not found: ID does not exist"
Nov 03 13:44:38.226623 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.226598    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"]
Nov 03 13:44:38.233925 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.233899    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8f8aa8-7db9fc4c8-86hcm"]
Nov 03 13:44:38.319436 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:38.319406    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" path="/var/lib/kubelet/pods/61c4cac3-b6f0-4f9f-b6fd-20105519b6c8/volumes"
Nov 03 13:44:40.690222 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:40.689739    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"]
Nov 03 13:44:40.690222 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:40.690042    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerName="istio-proxy" containerID="cri-o://961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8" gracePeriod=30
Nov 03 13:44:45.956286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:45.956225    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:44:46.154959 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.154926    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-podinfo\") pod \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") "
Nov 03 13:44:46.155157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.154975    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istiod-ca-cert\") pod \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") "
Nov 03 13:44:46.155157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155008    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-kube-api-access-x9jcs\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-kube-api-access-x9jcs\") pod \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") "
Nov 03 13:44:46.155157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155040    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-data\") pod \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") "
Nov 03 13:44:46.155157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155079    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-credential-socket\") pod \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") "
Nov 03 13:44:46.155157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155149    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-token\") pod \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") "
Nov 03 13:44:46.155470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155264    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-socket\") pod \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") "
Nov 03 13:44:46.155470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155306    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-certs\") pod \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") "
Nov 03 13:44:46.155470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155415    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-data" pod "4ddc3b97-c217-4a76-97cc-c0d6fab1c400" (UID: "4ddc3b97-c217-4a76-97cc-c0d6fab1c400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.155470 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155423    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-credential-socket" pod "4ddc3b97-c217-4a76-97cc-c0d6fab1c400" (UID: "4ddc3b97-c217-4a76-97cc-c0d6fab1c400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.155674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155588    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-socket" pod "4ddc3b97-c217-4a76-97cc-c0d6fab1c400" (UID: "4ddc3b97-c217-4a76-97cc-c0d6fab1c400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.155674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155609    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istiod-ca-cert" pod "4ddc3b97-c217-4a76-97cc-c0d6fab1c400" (UID: "4ddc3b97-c217-4a76-97cc-c0d6fab1c400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:46.155782 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155668    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-certs" pod "4ddc3b97-c217-4a76-97cc-c0d6fab1c400" (UID: "4ddc3b97-c217-4a76-97cc-c0d6fab1c400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.155782 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.155679    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-envoy\") pod \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\" (UID: \"4ddc3b97-c217-4a76-97cc-c0d6fab1c400\") "
Nov 03 13:44:46.156053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.156035    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.156053 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.156053    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.156180 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.156061    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.156180 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.156070    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.156180 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.156080    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.157358 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.157320    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-podinfo" pod "4ddc3b97-c217-4a76-97cc-c0d6fab1c400" (UID: "4ddc3b97-c217-4a76-97cc-c0d6fab1c400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:46.157477 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.157448    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-envoy" pod "4ddc3b97-c217-4a76-97cc-c0d6fab1c400" (UID: "4ddc3b97-c217-4a76-97cc-c0d6fab1c400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.157711 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.157685    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-token" pod "4ddc3b97-c217-4a76-97cc-c0d6fab1c400" (UID: "4ddc3b97-c217-4a76-97cc-c0d6fab1c400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:46.157711 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.157698    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-kube-api-access-x9jcs" pod "4ddc3b97-c217-4a76-97cc-c0d6fab1c400" (UID: "4ddc3b97-c217-4a76-97cc-c0d6fab1c400"). InnerVolumeSpecName "kube-api-access-x9jcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:46.256799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.256716    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.256799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.256743    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9jcs\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-kube-api-access-x9jcs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.256799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.256754    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.256799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.256765    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc3b97-c217-4a76-97cc-c0d6fab1c400-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.257732 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.257706    4447 generic.go:356] "Generic (PLEG): container finished" podID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerID="961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8" exitCode=0
Nov 03 13:44:46.257835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.257754    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"
Nov 03 13:44:46.257835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.257790    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" event={"ID":"4ddc3b97-c217-4a76-97cc-c0d6fab1c400","Type":"ContainerDied","Data":"961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8"}
Nov 03 13:44:46.257903 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.257846    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq" event={"ID":"4ddc3b97-c217-4a76-97cc-c0d6fab1c400","Type":"ContainerDied","Data":"a3a8d9ffb02420a4a40bf4977140ba14366963c2865dd4706df58d8c6499928a"}
Nov 03 13:44:46.257903 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.257872    4447 scope.go:122] "RemoveContainer" containerID="961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8"
Nov 03 13:44:46.267991 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.267975    4447 scope.go:122] "RemoveContainer" containerID="961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8"
Nov 03 13:44:46.268238 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:44:46.268216    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8\": container with ID starting with 961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8 not found: ID does not exist" containerID="961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8"
Nov 03 13:44:46.268288 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.268249    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8"} err="failed to get container status \"961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8\": rpc error: code = NotFound desc = could not find container \"961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8\": container with ID starting with 961234b748b47eac26ac17aeb7b9b4c65f1ec1e0f388b454083d8c946d666ce8 not found: ID does not exist"
Nov 03 13:44:46.287391 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.287367    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"]
Nov 03 13:44:46.304546 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.304525    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-qs9pq"]
Nov 03 13:44:46.318702 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:46.318681    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" path="/var/lib/kubelet/pods/4ddc3b97-c217-4a76-97cc-c0d6fab1c400/volumes"
Nov 03 13:44:50.292842 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:50.292807    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"]
Nov 03 13:44:55.293239 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:55.293165    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf" podUID="f56fb04c-d9fe-48e9-b361-2a93fb587f71" containerName="payload-pre-processing" containerID="cri-o://7c4036ac2cb6ef86cd5834dc5c7141198d9423b89e917cc8fb22136b01722852" gracePeriod=25
Nov 03 13:44:59.785304 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:59.785268    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"]
Nov 03 13:44:59.785787 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:59.785577    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8" podUID="024e3618-244e-4770-980b-1d3b0004341c" containerName="maas-api" containerID="cri-o://b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430" gracePeriod=30
Nov 03 13:44:59.869494 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:59.869457    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"]
Nov 03 13:44:59.869744 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:59.869704    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy" containerID="cri-o://4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3" gracePeriod=30
Nov 03 13:44:59.883568 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:59.883485    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"]
Nov 03 13:44:59.883996 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:44:59.883943    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerName="istio-proxy" containerID="cri-o://850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604" gracePeriod=30
Nov 03 13:45:00.086356 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.086304    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:45:00.172041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172006    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"]
Nov 03 13:45:00.172751 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172712    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy"
Nov 03 13:45:00.172815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172754    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy"
Nov 03 13:45:00.172815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172779    4447 cpu_manager.go:414] "containerMap: removing container" podUID="47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" containerName="payload-pre-processing"
Nov 03 13:45:00.172815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172787    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" containerName="payload-pre-processing"
Nov 03 13:45:00.172815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172804    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerName="istio-proxy"
Nov 03 13:45:00.172815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172811    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerName="istio-proxy"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172825    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172833    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172855    4447 cpu_manager.go:414] "containerMap: removing container" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172862    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172881    4447 cpu_manager.go:414] "containerMap: removing container" podUID="21a6efbf-968c-4e5d-93b3-a62eaae802d6" containerName="payload-processing"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172888    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a6efbf-968c-4e5d-93b3-a62eaae802d6" containerName="payload-processing"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172900    4447 cpu_manager.go:414] "containerMap: removing container" podUID="ca293290-1f26-4abf-a5ad-feccce1c7562" containerName="payload-processing"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172907    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca293290-1f26-4abf-a5ad-feccce1c7562" containerName="payload-processing"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172920    4447 cpu_manager.go:414] "containerMap: removing container" podUID="61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" containerName="maas-api"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172929    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" containerName="maas-api"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172940    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4b570f62-46d4-4b89-a7e1-9dc04c728f66" containerName="maas-api"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172947    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b570f62-46d4-4b89-a7e1-9dc04c728f66" containerName="maas-api"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172961    4447 cpu_manager.go:414] "containerMap: removing container" podUID="024e3618-244e-4770-980b-1d3b0004341c" containerName="maas-api"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172970    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024e3618-244e-4770-980b-1d3b0004341c" containerName="maas-api"
Nov 03 13:45:00.172980 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172984    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.172991    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173010    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e7405dbd-4d03-4519-b204-bdb453592985" containerName="payload-pre-processing"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173017    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7405dbd-4d03-4519-b204-bdb453592985" containerName="payload-pre-processing"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173026    4447 cpu_manager.go:414] "containerMap: removing container" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173041    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173054    4447 cpu_manager.go:414] "containerMap: removing container" podUID="da579797-607a-4132-945d-69c919276f0d" containerName="maas-api"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173061    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da579797-607a-4132-945d-69c919276f0d" containerName="maas-api"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173192    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a6efbf-968c-4e5d-93b3-a62eaae802d6" containerName="payload-processing"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173210    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca293290-1f26-4abf-a5ad-feccce1c7562" containerName="payload-processing"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173220    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c4cac3-b6f0-4f9f-b6fd-20105519b6c8" containerName="maas-api"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173229    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b570f62-46d4-4b89-a7e1-9dc04c728f66" containerName="maas-api"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173239    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="024e3618-244e-4770-980b-1d3b0004341c" containerName="maas-api"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173249    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b45341-138a-4e2e-a5bd-85fc060d7e17" containerName="istio-proxy"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173258    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7405dbd-4d03-4519-b204-bdb453592985" containerName="payload-pre-processing"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173266    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1416aa-d93f-4ed6-a950-32adb7bcae7f" containerName="istio-proxy"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173274    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="da579797-607a-4132-945d-69c919276f0d" containerName="maas-api"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173285    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde47a9b-1631-445e-bd1f-98e1da600c0a" containerName="istio-proxy"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173294    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cb9a4b-78ab-4a95-a88b-a9a2f83fac28" containerName="payload-pre-processing"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173305    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddc3b97-c217-4a76-97cc-c0d6fab1c400" containerName="istio-proxy"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173313    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0daabf9-3ca1-4963-baaa-ebcc1b3ee8e5" containerName="istio-proxy"
Nov 03 13:45:00.173517 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.173324    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="da62ab1b-1934-4d0b-84be-a9e1627b65e7" containerName="istio-proxy"
Nov 03 13:45:00.177534 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.177509    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:00.199662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.199631    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"]
Nov 03 13:45:00.210209 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.210183    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-metrics-certs\") pod \"024e3618-244e-4770-980b-1d3b0004341c\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") "
Nov 03 13:45:00.210371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.210276    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024e3618-244e-4770-980b-1d3b0004341c-kube-api-access-84nfj\" (UniqueName: \"kubernetes.io/projected/024e3618-244e-4770-980b-1d3b0004341c-kube-api-access-84nfj\") pod \"024e3618-244e-4770-980b-1d3b0004341c\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") "
Nov 03 13:45:00.210371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.210314    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-maas-api-tls\") pod \"024e3618-244e-4770-980b-1d3b0004341c\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") "
Nov 03 13:45:00.210514 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.210408    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024e3618-244e-4770-980b-1d3b0004341c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024e3618-244e-4770-980b-1d3b0004341c-openshift-service-ca\") pod \"024e3618-244e-4770-980b-1d3b0004341c\" (UID: \"024e3618-244e-4770-980b-1d3b0004341c\") "
Nov 03 13:45:00.210825 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.210796    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024e3618-244e-4770-980b-1d3b0004341c-openshift-service-ca" pod "024e3618-244e-4770-980b-1d3b0004341c" (UID: "024e3618-244e-4770-980b-1d3b0004341c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:00.211361 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.211321    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024e3618-244e-4770-980b-1d3b0004341c-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:00.212836 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.212805    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-metrics-certs" pod "024e3618-244e-4770-980b-1d3b0004341c" (UID: "024e3618-244e-4770-980b-1d3b0004341c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:45:00.213020 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.212986    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024e3618-244e-4770-980b-1d3b0004341c-kube-api-access-84nfj" pod "024e3618-244e-4770-980b-1d3b0004341c" (UID: "024e3618-244e-4770-980b-1d3b0004341c"). InnerVolumeSpecName "kube-api-access-84nfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:00.213609 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.213592    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-maas-api-tls" pod "024e3618-244e-4770-980b-1d3b0004341c" (UID: "024e3618-244e-4770-980b-1d3b0004341c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:45:00.290860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.290780    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:45:00.312498 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.312453    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895225-zt9k6\" (UID: \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:00.312674 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.312642    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ccr\" (UniqueName: \"kubernetes.io/projected/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-kube-api-access-26ccr\") pod \"maas-api-key-cleanup-29895225-zt9k6\" (UID: \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:00.312731 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.312713    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84nfj\" (UniqueName: \"kubernetes.io/projected/024e3618-244e-4770-980b-1d3b0004341c-kube-api-access-84nfj\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:00.312777 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.312761    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:00.312822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.312785    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024e3618-244e-4770-980b-1d3b0004341c-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:00.341725 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.341696    4447 generic.go:356] "Generic (PLEG): container finished" podID="024e3618-244e-4770-980b-1d3b0004341c" containerID="b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430" exitCode=0
Nov 03 13:45:00.341858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.341742    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"
Nov 03 13:45:00.341858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.341785    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8" event={"ID":"024e3618-244e-4770-980b-1d3b0004341c","Type":"ContainerDied","Data":"b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430"}
Nov 03 13:45:00.341858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.341833    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8" event={"ID":"024e3618-244e-4770-980b-1d3b0004341c","Type":"ContainerDied","Data":"622dd5d215c970296fdde148dbf1b2bb3cdacf4e101d63d023fa5b64cdad0725"}
Nov 03 13:45:00.341858 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.341854    4447 scope.go:122] "RemoveContainer" containerID="b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430"
Nov 03 13:45:00.351906 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.351881    4447 scope.go:122] "RemoveContainer" containerID="b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430"
Nov 03 13:45:00.352245 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:45:00.352227    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430\": container with ID starting with b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430 not found: ID does not exist" containerID="b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430"
Nov 03 13:45:00.352382 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.352256    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430"} err="failed to get container status \"b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430\": rpc error: code = NotFound desc = could not find container \"b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430\": container with ID starting with b9a506f95ac6f33205738a6fc55fe54c40ee4d0274cdbf65627f0c9ba6566430 not found: ID does not exist"
Nov 03 13:45:00.377469 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.377434    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"]
Nov 03 13:45:00.383286 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.383257    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-08111b-7bb65c959-9mzh8"]
Nov 03 13:45:00.414224 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.413793    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ccr\" (UniqueName: \"kubernetes.io/projected/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-kube-api-access-26ccr\") pod \"maas-api-key-cleanup-29895225-zt9k6\" (UID: \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:00.414224 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.413912    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895225-zt9k6\" (UID: \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:00.414733 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.414706    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895225-zt9k6\" (UID: \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:00.430188 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.430134    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ccr\" (UniqueName: \"kubernetes.io/projected/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-kube-api-access-26ccr\") pod \"maas-api-key-cleanup-29895225-zt9k6\" (UID: \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:00.496557 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.496518    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:00.787516 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.787479    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"]
Nov 03 13:45:00.789733 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:45:00.789708    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98f16b0_a1ca_4d2e_8f57_bd5822efba24.slice/crio-051c3ef8a05551c9090c52d82fd569803b93fae391cfe6fd41530d03213d8b8b WatchSource:0}: Error finding container 051c3ef8a05551c9090c52d82fd569803b93fae391cfe6fd41530d03213d8b8b: Status 404 returned error can't find the container with id 051c3ef8a05551c9090c52d82fd569803b93fae391cfe6fd41530d03213d8b8b
Nov 03 13:45:00.806771 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.806746    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf" podUID="f56fb04c-d9fe-48e9-b361-2a93fb587f71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:00.841587 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.841554    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"]
Nov 03 13:45:00.846279 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.846255    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:00.866792 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.866758    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:45:00.870084 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.870058    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"]
Nov 03 13:45:00.919606 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.919574    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzvxr\" (UniqueName: \"kubernetes.io/projected/3f740758-1855-4142-8a32-66d78965577c-kube-api-access-gzvxr\") pod \"maas-api-revoke-sub-c224e916-ng6wz\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:00.919790 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.919699    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f740758-1855-4142-8a32-66d78965577c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c224e916-ng6wz\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:00.919790 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:00.919765    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f740758-1855-4142-8a32-66d78965577c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c224e916-ng6wz\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:01.021306 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.021264    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzvxr\" (UniqueName: \"kubernetes.io/projected/3f740758-1855-4142-8a32-66d78965577c-kube-api-access-gzvxr\") pod \"maas-api-revoke-sub-c224e916-ng6wz\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:01.021550 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.021384    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f740758-1855-4142-8a32-66d78965577c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c224e916-ng6wz\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:01.021550 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.021428    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f740758-1855-4142-8a32-66d78965577c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c224e916-ng6wz\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:01.022071 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.022046    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f740758-1855-4142-8a32-66d78965577c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c224e916-ng6wz\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:01.023870 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.023843    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f740758-1855-4142-8a32-66d78965577c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c224e916-ng6wz\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:01.042240 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.037893    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzvxr\" (UniqueName: \"kubernetes.io/projected/3f740758-1855-4142-8a32-66d78965577c-kube-api-access-gzvxr\") pod \"maas-api-revoke-sub-c224e916-ng6wz\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:01.159219 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.159141    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:01.350987 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.350952    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" event={"ID":"d98f16b0-a1ca-4d2e-8f57-bd5822efba24","Type":"ContainerStarted","Data":"5b6ecf174ecce477277e26bff11af729cc183c63fdcdc77879afa59d2df99d4d"}
Nov 03 13:45:01.351195 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.351004    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" event={"ID":"d98f16b0-a1ca-4d2e-8f57-bd5822efba24","Type":"ContainerStarted","Data":"051c3ef8a05551c9090c52d82fd569803b93fae391cfe6fd41530d03213d8b8b"}
Nov 03 13:45:01.384062 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.383960    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" podStartSLOduration=1.383940231 podStartE2EDuration="1.383940231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:01.383920565 +0000 UTC m=+2189.195588754" watchObservedRunningTime="2026-11-03 13:45:01.383940231 +0000 UTC m=+2189.195608424"
Nov 03 13:45:01.461711 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:01.461676    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"]
Nov 03 13:45:01.461931 ip-10-0-132-167 kubenswrapper[4447]: W1103 13:45:01.461911    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f740758_1855_4142_8a32_66d78965577c.slice/crio-a038a55b7d522845d3b3f4a9fd61667248307cfed7b19b121c2f7e07c5e7e932 WatchSource:0}: Error finding container a038a55b7d522845d3b3f4a9fd61667248307cfed7b19b121c2f7e07c5e7e932: Status 404 returned error can't find the container with id a038a55b7d522845d3b3f4a9fd61667248307cfed7b19b121c2f7e07c5e7e932
Nov 03 13:45:02.318974 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:02.318942    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024e3618-244e-4770-980b-1d3b0004341c" path="/var/lib/kubelet/pods/024e3618-244e-4770-980b-1d3b0004341c/volumes"
Nov 03 13:45:02.361766 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:02.361730    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz" event={"ID":"3f740758-1855-4142-8a32-66d78965577c","Type":"ContainerStarted","Data":"31ecebba842fa08c7694ab2be202c3dace76f2e607cd8982aea44d041251c76f"}
Nov 03 13:45:02.361932 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:02.361786    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz" event={"ID":"3f740758-1855-4142-8a32-66d78965577c","Type":"ContainerStarted","Data":"a038a55b7d522845d3b3f4a9fd61667248307cfed7b19b121c2f7e07c5e7e932"}
Nov 03 13:45:02.363665 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:02.363632    4447 generic.go:356] "Generic (PLEG): container finished" podID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerID="5b6ecf174ecce477277e26bff11af729cc183c63fdcdc77879afa59d2df99d4d" exitCode=60
Nov 03 13:45:02.363805 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:02.363715    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" event={"ID":"d98f16b0-a1ca-4d2e-8f57-bd5822efba24","Type":"ContainerDied","Data":"5b6ecf174ecce477277e26bff11af729cc183c63fdcdc77879afa59d2df99d4d"}
Nov 03 13:45:02.363926 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:02.363910    4447 scope.go:122] "RemoveContainer" containerID="5b6ecf174ecce477277e26bff11af729cc183c63fdcdc77879afa59d2df99d4d"
Nov 03 13:45:02.443131 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:02.443078    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz" podStartSLOduration=2.443061888 podStartE2EDuration="2.443061888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:02.402012869 +0000 UTC m=+2190.213681073" watchObservedRunningTime="2026-11-03 13:45:02.443061888 +0000 UTC m=+2190.254730077"
Nov 03 13:45:03.371528 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:03.371491    4447 generic.go:356] "Generic (PLEG): container finished" podID="3f740758-1855-4142-8a32-66d78965577c" containerID="31ecebba842fa08c7694ab2be202c3dace76f2e607cd8982aea44d041251c76f" exitCode=0
Nov 03 13:45:03.371978 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:03.371577    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz" event={"ID":"3f740758-1855-4142-8a32-66d78965577c","Type":"ContainerDied","Data":"31ecebba842fa08c7694ab2be202c3dace76f2e607cd8982aea44d041251c76f"}
Nov 03 13:45:03.373541 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:03.373516    4447 generic.go:356] "Generic (PLEG): container finished" podID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerID="bf89b0fa76634beedf8e924e8fe9be6d2e546f2080d3415f67e46d4929f199d8" exitCode=60
Nov 03 13:45:03.373689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:03.373561    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" event={"ID":"d98f16b0-a1ca-4d2e-8f57-bd5822efba24","Type":"ContainerDied","Data":"bf89b0fa76634beedf8e924e8fe9be6d2e546f2080d3415f67e46d4929f199d8"}
Nov 03 13:45:03.373689 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:03.373602    4447 scope.go:122] "RemoveContainer" containerID="5b6ecf174ecce477277e26bff11af729cc183c63fdcdc77879afa59d2df99d4d"
Nov 03 13:45:03.373917 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:03.373902    4447 scope.go:122] "RemoveContainer" containerID="bf89b0fa76634beedf8e924e8fe9be6d2e546f2080d3415f67e46d4929f199d8"
Nov 03 13:45:03.374197 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:45:03.374178    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895225-zt9k6_odh-ai-gateway-infra(d98f16b0-a1ca-4d2e-8f57-bd5822efba24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24"
Nov 03 13:45:03.416255 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:45:03.416214    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:45:04.381941 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.381691    4447 scope.go:122] "RemoveContainer" containerID="bf89b0fa76634beedf8e924e8fe9be6d2e546f2080d3415f67e46d4929f199d8"
Nov 03 13:45:04.382159 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:45:04.382016    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895225-zt9k6_odh-ai-gateway-infra(d98f16b0-a1ca-4d2e-8f57-bd5822efba24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24"
Nov 03 13:45:04.527621 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.527593    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:04.677639 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.677556    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f740758-1855-4142-8a32-66d78965577c-kube-api-access-gzvxr\" (UniqueName: \"kubernetes.io/projected/3f740758-1855-4142-8a32-66d78965577c-kube-api-access-gzvxr\") pod \"3f740758-1855-4142-8a32-66d78965577c\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") "
Nov 03 13:45:04.677823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.677665    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f740758-1855-4142-8a32-66d78965577c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f740758-1855-4142-8a32-66d78965577c-maas-api-xks-ca\") pod \"3f740758-1855-4142-8a32-66d78965577c\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") "
Nov 03 13:45:04.677823 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.677717    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f740758-1855-4142-8a32-66d78965577c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f740758-1855-4142-8a32-66d78965577c-maas-api-openshift-ca\") pod \"3f740758-1855-4142-8a32-66d78965577c\" (UID: \"3f740758-1855-4142-8a32-66d78965577c\") "
Nov 03 13:45:04.678123 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.678093    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f740758-1855-4142-8a32-66d78965577c-maas-api-openshift-ca" pod "3f740758-1855-4142-8a32-66d78965577c" (UID: "3f740758-1855-4142-8a32-66d78965577c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:04.679835 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.679811    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f740758-1855-4142-8a32-66d78965577c-maas-api-xks-ca" pod "3f740758-1855-4142-8a32-66d78965577c" (UID: "3f740758-1855-4142-8a32-66d78965577c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:45:04.679942 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.679820    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f740758-1855-4142-8a32-66d78965577c-kube-api-access-gzvxr" pod "3f740758-1855-4142-8a32-66d78965577c" (UID: "3f740758-1855-4142-8a32-66d78965577c"). InnerVolumeSpecName "kube-api-access-gzvxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:04.779061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.779026    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f740758-1855-4142-8a32-66d78965577c-maas-api-xks-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:04.779061 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.779059    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f740758-1855-4142-8a32-66d78965577c-maas-api-openshift-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:04.779272 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:04.779073    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzvxr\" (UniqueName: \"kubernetes.io/projected/3f740758-1855-4142-8a32-66d78965577c-kube-api-access-gzvxr\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.152065 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.152003    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:45:05.222047 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.222016    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:45:05.288253 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288218    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-certs\") pod \"aaaf70db-dc8c-451f-a584-77b665a67fd3\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") "
Nov 03 13:45:05.288450 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288263    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaaf70db-dc8c-451f-a584-77b665a67fd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaf70db-dc8c-451f-a584-77b665a67fd3-istiod-ca-cert\") pod \"aaaf70db-dc8c-451f-a584-77b665a67fd3\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") "
Nov 03 13:45:05.288450 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288327    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-credential-socket\") pod \"aaaf70db-dc8c-451f-a584-77b665a67fd3\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") "
Nov 03 13:45:05.288450 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288367    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-envoy\") pod \"aaaf70db-dc8c-451f-a584-77b665a67fd3\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") "
Nov 03 13:45:05.288450 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288421    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-data\") pod \"aaaf70db-dc8c-451f-a584-77b665a67fd3\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") "
Nov 03 13:45:05.288662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288457    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-certs\") pod \"92849677-985f-4a8f-b812-bd24d20b7c8a\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") "
Nov 03 13:45:05.288662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288498    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-token\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-token\") pod \"aaaf70db-dc8c-451f-a584-77b665a67fd3\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") "
Nov 03 13:45:05.288662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288572    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-certs" pod "aaaf70db-dc8c-451f-a584-77b665a67fd3" (UID: "aaaf70db-dc8c-451f-a584-77b665a67fd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.288819 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288682    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-credential-socket" pod "aaaf70db-dc8c-451f-a584-77b665a67fd3" (UID: "aaaf70db-dc8c-451f-a584-77b665a67fd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.288819 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288698    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-certs" pod "92849677-985f-4a8f-b812-bd24d20b7c8a" (UID: "92849677-985f-4a8f-b812-bd24d20b7c8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.288819 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288786    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-data" pod "aaaf70db-dc8c-451f-a584-77b665a67fd3" (UID: "aaaf70db-dc8c-451f-a584-77b665a67fd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.288929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288834    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaaf70db-dc8c-451f-a584-77b665a67fd3-istiod-ca-cert" pod "aaaf70db-dc8c-451f-a584-77b665a67fd3" (UID: "aaaf70db-dc8c-451f-a584-77b665a67fd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:05.288929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288911    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-podinfo\") pod \"92849677-985f-4a8f-b812-bd24d20b7c8a\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") "
Nov 03 13:45:05.289003 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288949    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-data\") pod \"92849677-985f-4a8f-b812-bd24d20b7c8a\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") "
Nov 03 13:45:05.289003 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.288976    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-credential-socket\") pod \"92849677-985f-4a8f-b812-bd24d20b7c8a\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") "
Nov 03 13:45:05.289117 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289005    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-socket\") pod \"aaaf70db-dc8c-451f-a584-77b665a67fd3\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") "
Nov 03 13:45:05.289117 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289053    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-kube-api-access-qlz75\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-kube-api-access-qlz75\") pod \"92849677-985f-4a8f-b812-bd24d20b7c8a\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") "
Nov 03 13:45:05.289117 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289083    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-socket\") pod \"92849677-985f-4a8f-b812-bd24d20b7c8a\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") "
Nov 03 13:45:05.289266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289134    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92849677-985f-4a8f-b812-bd24d20b7c8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92849677-985f-4a8f-b812-bd24d20b7c8a-istiod-ca-cert\") pod \"92849677-985f-4a8f-b812-bd24d20b7c8a\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") "
Nov 03 13:45:05.289266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289232    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-envoy\") pod \"92849677-985f-4a8f-b812-bd24d20b7c8a\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") "
Nov 03 13:45:05.289266 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289238    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-data" pod "92849677-985f-4a8f-b812-bd24d20b7c8a" (UID: "92849677-985f-4a8f-b812-bd24d20b7c8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.289448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289293    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-kube-api-access-kzdlc\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-kube-api-access-kzdlc\") pod \"aaaf70db-dc8c-451f-a584-77b665a67fd3\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") "
Nov 03 13:45:05.289448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289319    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-podinfo\") pod \"aaaf70db-dc8c-451f-a584-77b665a67fd3\" (UID: \"aaaf70db-dc8c-451f-a584-77b665a67fd3\") "
Nov 03 13:45:05.289448 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289357    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-token\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-token\") pod \"92849677-985f-4a8f-b812-bd24d20b7c8a\" (UID: \"92849677-985f-4a8f-b812-bd24d20b7c8a\") "
Nov 03 13:45:05.289598 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289479    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-credential-socket" pod "92849677-985f-4a8f-b812-bd24d20b7c8a" (UID: "92849677-985f-4a8f-b812-bd24d20b7c8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.289598 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289493    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-socket" pod "92849677-985f-4a8f-b812-bd24d20b7c8a" (UID: "92849677-985f-4a8f-b812-bd24d20b7c8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.289804 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289785    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.289864 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289810    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.289864 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289826    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaf70db-dc8c-451f-a584-77b665a67fd3-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.289864 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289839    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.289864 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289853    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.290080 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289867    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.290080 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289881    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.290080 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.289895    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.290233 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.290103    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-socket" pod "aaaf70db-dc8c-451f-a584-77b665a67fd3" (UID: "aaaf70db-dc8c-451f-a584-77b665a67fd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.290298 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.290227    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92849677-985f-4a8f-b812-bd24d20b7c8a-istiod-ca-cert" pod "92849677-985f-4a8f-b812-bd24d20b7c8a" (UID: "92849677-985f-4a8f-b812-bd24d20b7c8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:05.291157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.291129    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-envoy" pod "aaaf70db-dc8c-451f-a584-77b665a67fd3" (UID: "aaaf70db-dc8c-451f-a584-77b665a67fd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.291455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.291430    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-podinfo" pod "92849677-985f-4a8f-b812-bd24d20b7c8a" (UID: "92849677-985f-4a8f-b812-bd24d20b7c8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:45:05.291737 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.291709    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-token" pod "aaaf70db-dc8c-451f-a584-77b665a67fd3" (UID: "aaaf70db-dc8c-451f-a584-77b665a67fd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:05.291929 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.291910    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-token" pod "92849677-985f-4a8f-b812-bd24d20b7c8a" (UID: "92849677-985f-4a8f-b812-bd24d20b7c8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:05.292092 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.292076    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-envoy" pod "92849677-985f-4a8f-b812-bd24d20b7c8a" (UID: "92849677-985f-4a8f-b812-bd24d20b7c8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:05.292261 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.292244    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-kube-api-access-qlz75" pod "92849677-985f-4a8f-b812-bd24d20b7c8a" (UID: "92849677-985f-4a8f-b812-bd24d20b7c8a"). InnerVolumeSpecName "kube-api-access-qlz75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:05.292562 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.292543    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-podinfo" pod "aaaf70db-dc8c-451f-a584-77b665a67fd3" (UID: "aaaf70db-dc8c-451f-a584-77b665a67fd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:45:05.292604 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.292555    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-kube-api-access-kzdlc" pod "aaaf70db-dc8c-451f-a584-77b665a67fd3" (UID: "aaaf70db-dc8c-451f-a584-77b665a67fd3"). InnerVolumeSpecName "kube-api-access-kzdlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:05.389037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.388991    4447 generic.go:356] "Generic (PLEG): container finished" podID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerID="850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604" exitCode=0
Nov 03 13:45:05.389574 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.389050    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"
Nov 03 13:45:05.389574 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.389081    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" event={"ID":"92849677-985f-4a8f-b812-bd24d20b7c8a","Type":"ContainerDied","Data":"850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604"}
Nov 03 13:45:05.389574 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.389169    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6" event={"ID":"92849677-985f-4a8f-b812-bd24d20b7c8a","Type":"ContainerDied","Data":"bd2af8cd6e3f376140d261aa439bbe5fb11d3f6b140b26790ab715e23b39f916"}
Nov 03 13:45:05.389574 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.389203    4447 scope.go:122] "RemoveContainer" containerID="850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604"
Nov 03 13:45:05.390607 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390567    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390607 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390589    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390607 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390602    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390615    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaf70db-dc8c-451f-a584-77b665a67fd3-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390630    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlz75\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-kube-api-access-qlz75\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390641    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92849677-985f-4a8f-b812-bd24d20b7c8a-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390652    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390664    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzdlc\" (UniqueName: \"kubernetes.io/projected/aaaf70db-dc8c-451f-a584-77b665a67fd3-kube-api-access-kzdlc\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390677    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaf70db-dc8c-451f-a584-77b665a67fd3-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390690    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92849677-985f-4a8f-b812-bd24d20b7c8a-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390727    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz" event={"ID":"3f740758-1855-4142-8a32-66d78965577c","Type":"ContainerDied","Data":"a038a55b7d522845d3b3f4a9fd61667248307cfed7b19b121c2f7e07c5e7e932"}
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390749    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"
Nov 03 13:45:05.390785 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.390771    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a038a55b7d522845d3b3f4a9fd61667248307cfed7b19b121c2f7e07c5e7e932"
Nov 03 13:45:05.393759 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.393733    4447 generic.go:356] "Generic (PLEG): container finished" podID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerID="4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3" exitCode=0
Nov 03 13:45:05.393759 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.393767    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"
Nov 03 13:45:05.393935 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.393795    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" event={"ID":"aaaf70db-dc8c-451f-a584-77b665a67fd3","Type":"ContainerDied","Data":"4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3"}
Nov 03 13:45:05.393935 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.393838    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6" event={"ID":"aaaf70db-dc8c-451f-a584-77b665a67fd3","Type":"ContainerDied","Data":"8250e1504bc1125e4360d8298bca3109f53aa4e6b18b45e0b9e0574bb7687ba2"}
Nov 03 13:45:05.401877 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.401850    4447 scope.go:122] "RemoveContainer" containerID="850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604"
Nov 03 13:45:05.402249 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:45:05.402136    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604\": container with ID starting with 850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604 not found: ID does not exist" containerID="850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604"
Nov 03 13:45:05.402249 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.402162    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604"} err="failed to get container status \"850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604\": rpc error: code = NotFound desc = could not find container \"850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604\": container with ID starting with 850554012b127c529caf88018e44994939e921c8b9617316467925be5ff87604 not found: ID does not exist"
Nov 03 13:45:05.402249 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.402179    4447 scope.go:122] "RemoveContainer" containerID="4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3"
Nov 03 13:45:05.411860 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.411841    4447 scope.go:122] "RemoveContainer" containerID="4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3"
Nov 03 13:45:05.412113 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:45:05.412090    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3\": container with ID starting with 4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3 not found: ID does not exist" containerID="4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3"
Nov 03 13:45:05.412176 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.412122    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3"} err="failed to get container status \"4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3\": rpc error: code = NotFound desc = could not find container \"4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3\": container with ID starting with 4c33d085a676ffa5c530d587aafc2417f94e0062bcaf4ddcaf8f778ee23e5de3 not found: ID does not exist"
Nov 03 13:45:05.431762 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.431728    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"]
Nov 03 13:45:05.434248 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.434227    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-224r6"]
Nov 03 13:45:05.466837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.466755    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"]
Nov 03 13:45:05.469057 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:05.469019    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-tw9v6"]
Nov 03 13:45:06.319285 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:06.319250    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" path="/var/lib/kubelet/pods/92849677-985f-4a8f-b812-bd24d20b7c8a/volumes"
Nov 03 13:45:06.319777 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:06.319763    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" path="/var/lib/kubelet/pods/aaaf70db-dc8c-451f-a584-77b665a67fd3/volumes"
Nov 03 13:45:10.806728 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:10.806685    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf" podUID="f56fb04c-d9fe-48e9-b361-2a93fb587f71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:14.313089 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:14.313050    4447 scope.go:122] "RemoveContainer" containerID="bf89b0fa76634beedf8e924e8fe9be6d2e546f2080d3415f67e46d4929f199d8"
Nov 03 13:45:15.457045 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.456999    4447 generic.go:356] "Generic (PLEG): container finished" podID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerID="8503a9469bc10f5a56faf5a1d03927bb5f41d1e8bc3e3bee4c244e19128a77d5" exitCode=60
Nov 03 13:45:15.457527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.457071    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" event={"ID":"d98f16b0-a1ca-4d2e-8f57-bd5822efba24","Type":"ContainerDied","Data":"8503a9469bc10f5a56faf5a1d03927bb5f41d1e8bc3e3bee4c244e19128a77d5"}
Nov 03 13:45:15.457527 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.457145    4447 scope.go:122] "RemoveContainer" containerID="bf89b0fa76634beedf8e924e8fe9be6d2e546f2080d3415f67e46d4929f199d8"
Nov 03 13:45:15.457731 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.457568    4447 scope.go:122] "RemoveContainer" containerID="8503a9469bc10f5a56faf5a1d03927bb5f41d1e8bc3e3bee4c244e19128a77d5"
Nov 03 13:45:15.458190 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:45:15.458087    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895225-zt9k6_odh-ai-gateway-infra(d98f16b0-a1ca-4d2e-8f57-bd5822efba24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24"
Nov 03 13:45:15.459822 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.459803    4447 generic.go:356] "Generic (PLEG): container finished" podID="f56fb04c-d9fe-48e9-b361-2a93fb587f71" containerID="7c4036ac2cb6ef86cd5834dc5c7141198d9423b89e917cc8fb22136b01722852" exitCode=0
Nov 03 13:45:15.459908 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.459874    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf" event={"ID":"f56fb04c-d9fe-48e9-b361-2a93fb587f71","Type":"ContainerDied","Data":"7c4036ac2cb6ef86cd5834dc5c7141198d9423b89e917cc8fb22136b01722852"}
Nov 03 13:45:15.557411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.557380    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:45:15.704590 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.704559    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56fb04c-d9fe-48e9-b361-2a93fb587f71-kube-api-access-dt4j4\" (UniqueName: \"kubernetes.io/projected/f56fb04c-d9fe-48e9-b361-2a93fb587f71-kube-api-access-dt4j4\") pod \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\" (UID: \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\") "
Nov 03 13:45:15.704762 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.704668    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56fb04c-d9fe-48e9-b361-2a93fb587f71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56fb04c-d9fe-48e9-b361-2a93fb587f71-plugins-config-volume\") pod \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\" (UID: \"f56fb04c-d9fe-48e9-b361-2a93fb587f71\") "
Nov 03 13:45:15.705082 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.705051    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56fb04c-d9fe-48e9-b361-2a93fb587f71-plugins-config-volume" pod "f56fb04c-d9fe-48e9-b361-2a93fb587f71" (UID: "f56fb04c-d9fe-48e9-b361-2a93fb587f71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:15.706847 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.706826    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56fb04c-d9fe-48e9-b361-2a93fb587f71-kube-api-access-dt4j4" pod "f56fb04c-d9fe-48e9-b361-2a93fb587f71" (UID: "f56fb04c-d9fe-48e9-b361-2a93fb587f71"). InnerVolumeSpecName "kube-api-access-dt4j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:15.806002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.805911    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt4j4\" (UniqueName: \"kubernetes.io/projected/f56fb04c-d9fe-48e9-b361-2a93fb587f71-kube-api-access-dt4j4\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:15.806002 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:15.805950    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56fb04c-d9fe-48e9-b361-2a93fb587f71-plugins-config-volume\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:16.475354 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.475312    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"
Nov 03 13:45:16.475354 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.475326    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf" event={"ID":"f56fb04c-d9fe-48e9-b361-2a93fb587f71","Type":"ContainerDied","Data":"997c7af65fa6cd0d63f4aab2bcc0d2a49bbdfc1d04c0874040cc66d3844ce7f7"}
Nov 03 13:45:16.475868 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.475432    4447 scope.go:122] "RemoveContainer" containerID="7c4036ac2cb6ef86cd5834dc5c7141198d9423b89e917cc8fb22136b01722852"
Nov 03 13:45:16.491439 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.491412    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"]
Nov 03 13:45:16.496034 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.496012    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"]
Nov 03 13:45:16.502652 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.502626    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9add72-67bbb65659-mb9gf"]
Nov 03 13:45:16.653770 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.653739    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:16.715636 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.715603    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-kube-api-access-26ccr\" (UniqueName: \"kubernetes.io/projected/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-kube-api-access-26ccr\") pod \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\" (UID: \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\") "
Nov 03 13:45:16.715783 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.715647    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-maas-api-service-ca\") pod \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\" (UID: \"d98f16b0-a1ca-4d2e-8f57-bd5822efba24\") "
Nov 03 13:45:16.716037 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.716016    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-maas-api-service-ca" pod "d98f16b0-a1ca-4d2e-8f57-bd5822efba24" (UID: "d98f16b0-a1ca-4d2e-8f57-bd5822efba24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:16.717824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.717796    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-kube-api-access-26ccr" pod "d98f16b0-a1ca-4d2e-8f57-bd5822efba24" (UID: "d98f16b0-a1ca-4d2e-8f57-bd5822efba24"). InnerVolumeSpecName "kube-api-access-26ccr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:16.816570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.816521    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26ccr\" (UniqueName: \"kubernetes.io/projected/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-kube-api-access-26ccr\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:16.816570 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:16.816562    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98f16b0-a1ca-4d2e-8f57-bd5822efba24-maas-api-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:17.482378 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:17.482332    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"
Nov 03 13:45:17.482378 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:17.482332    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6" event={"ID":"d98f16b0-a1ca-4d2e-8f57-bd5822efba24","Type":"ContainerDied","Data":"051c3ef8a05551c9090c52d82fd569803b93fae391cfe6fd41530d03213d8b8b"}
Nov 03 13:45:17.482938 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:17.482425    4447 scope.go:122] "RemoveContainer" containerID="8503a9469bc10f5a56faf5a1d03927bb5f41d1e8bc3e3bee4c244e19128a77d5"
Nov 03 13:45:17.507935 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:17.507908    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"]
Nov 03 13:45:17.513107 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:17.513080    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-zt9k6"]
Nov 03 13:45:18.318872 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:18.318835    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" path="/var/lib/kubelet/pods/d98f16b0-a1ca-4d2e-8f57-bd5822efba24/volumes"
Nov 03 13:45:18.319263 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:18.319250    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56fb04c-d9fe-48e9-b361-2a93fb587f71" path="/var/lib/kubelet/pods/f56fb04c-d9fe-48e9-b361-2a93fb587f71/volumes"
Nov 03 13:45:33.414404 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:45:33.414327    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:45:50.628758 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:50.628724    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"]
Nov 03 13:45:50.629246 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:50.628936    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy" containerID="cri-o://857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69" gracePeriod=30
Nov 03 13:45:54.484628 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:54.484578    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:45:54.485019 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:54.484662    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:45:55.902562 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.902501    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:45:55.951605 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.951569    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-istio-token\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-istio-token\") pod \"9f16b044-d645-494f-a857-00264f18ce26\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") "
Nov 03 13:45:55.951794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.951649    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-credential-socket\") pod \"9f16b044-d645-494f-a857-00264f18ce26\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") "
Nov 03 13:45:55.951794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.951691    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-envoy\") pod \"9f16b044-d645-494f-a857-00264f18ce26\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") "
Nov 03 13:45:55.951794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.951742    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-certs\") pod \"9f16b044-d645-494f-a857-00264f18ce26\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") "
Nov 03 13:45:55.951794 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.951792    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-kube-api-access-862fb\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-kube-api-access-862fb\") pod \"9f16b044-d645-494f-a857-00264f18ce26\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") "
Nov 03 13:45:55.952026 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.951849    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f16b044-d645-494f-a857-00264f18ce26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f16b044-d645-494f-a857-00264f18ce26-istiod-ca-cert\") pod \"9f16b044-d645-494f-a857-00264f18ce26\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") "
Nov 03 13:45:55.952026 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.951906    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f16b044-d645-494f-a857-00264f18ce26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f16b044-d645-494f-a857-00264f18ce26-istio-podinfo\") pod \"9f16b044-d645-494f-a857-00264f18ce26\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") "
Nov 03 13:45:55.952026 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.951955    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-data\") pod \"9f16b044-d645-494f-a857-00264f18ce26\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") "
Nov 03 13:45:55.952026 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.951980    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-socket\") pod \"9f16b044-d645-494f-a857-00264f18ce26\" (UID: \"9f16b044-d645-494f-a857-00264f18ce26\") "
Nov 03 13:45:55.952250 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.952001    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-credential-socket" pod "9f16b044-d645-494f-a857-00264f18ce26" (UID: "9f16b044-d645-494f-a857-00264f18ce26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:55.952250 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.952080    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-certs" pod "9f16b044-d645-494f-a857-00264f18ce26" (UID: "9f16b044-d645-494f-a857-00264f18ce26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:55.952400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.952306    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-data" pod "9f16b044-d645-494f-a857-00264f18ce26" (UID: "9f16b044-d645-494f-a857-00264f18ce26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:55.952400 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.952303    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f16b044-d645-494f-a857-00264f18ce26-istiod-ca-cert" pod "9f16b044-d645-494f-a857-00264f18ce26" (UID: "9f16b044-d645-494f-a857-00264f18ce26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:55.952523 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.952431    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:55.952523 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.952451    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:55.952523 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.952466    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:55.952683 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.952544    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-socket" pod "9f16b044-d645-494f-a857-00264f18ce26" (UID: "9f16b044-d645-494f-a857-00264f18ce26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:55.954235 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.954212    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f16b044-d645-494f-a857-00264f18ce26-istio-podinfo" pod "9f16b044-d645-494f-a857-00264f18ce26" (UID: "9f16b044-d645-494f-a857-00264f18ce26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:45:55.954435 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.954416    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-envoy" pod "9f16b044-d645-494f-a857-00264f18ce26" (UID: "9f16b044-d645-494f-a857-00264f18ce26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:55.954637 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.954616    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-kube-api-access-862fb" pod "9f16b044-d645-494f-a857-00264f18ce26" (UID: "9f16b044-d645-494f-a857-00264f18ce26"). InnerVolumeSpecName "kube-api-access-862fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:55.955575 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:55.955553    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-istio-token" pod "9f16b044-d645-494f-a857-00264f18ce26" (UID: "9f16b044-d645-494f-a857-00264f18ce26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:56.053739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.053707    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f16b044-d645-494f-a857-00264f18ce26-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:56.053739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.053736    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:56.053739 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.053745    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:56.053988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.053756    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f16b044-d645-494f-a857-00264f18ce26-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:56.053988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.053766    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-862fb\" (UniqueName: \"kubernetes.io/projected/9f16b044-d645-494f-a857-00264f18ce26-kube-api-access-862fb\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:56.053988 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.053776    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f16b044-d645-494f-a857-00264f18ce26-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:56.718645 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.718606    4447 generic.go:356] "Generic (PLEG): container finished" podID="9f16b044-d645-494f-a857-00264f18ce26" containerID="857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69" exitCode=0
Nov 03 13:45:56.718824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.718676    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"
Nov 03 13:45:56.718824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.718689    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" event={"ID":"9f16b044-d645-494f-a857-00264f18ce26","Type":"ContainerDied","Data":"857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69"}
Nov 03 13:45:56.718824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.718770    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff" event={"ID":"9f16b044-d645-494f-a857-00264f18ce26","Type":"ContainerDied","Data":"6d86c4b86bade7c4e23206690b6ae4cd80278ce56b4e4a97d018dab3b44a42bd"}
Nov 03 13:45:56.718824 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.718792    4447 scope.go:122] "RemoveContainer" containerID="857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69"
Nov 03 13:45:56.728237 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.728215    4447 scope.go:122] "RemoveContainer" containerID="857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69"
Nov 03 13:45:56.728524 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:45:56.728505    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69\": container with ID starting with 857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69 not found: ID does not exist" containerID="857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69"
Nov 03 13:45:56.728598 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.728532    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69"} err="failed to get container status \"857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69\": rpc error: code = NotFound desc = could not find container \"857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69\": container with ID starting with 857a4f9f86cef7f6ca0a850206fd8cf27ea134fffcd79c8beff19ea45a4a5a69 not found: ID does not exist"
Nov 03 13:45:56.739626 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.739579    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"]
Nov 03 13:45:56.742117 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:56.742082    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-z7bff"]
Nov 03 13:45:58.321215 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:58.321178    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f16b044-d645-494f-a857-00264f18ce26" path="/var/lib/kubelet/pods/9f16b044-d645-494f-a857-00264f18ce26/volumes"
Nov 03 13:45:58.347096 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:58.347063    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"]
Nov 03 13:45:58.347302 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:45:58.347276    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy" containerID="cri-o://be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545" gracePeriod=30
Nov 03 13:46:01.781371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:01.781332    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-tbkh6_d5388bcf-9195-4a03-89b5-d50dcb20b89d/manager/0.log"
Nov 03 13:46:01.974395 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:01.974363    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8w2nb_1ab4053b-cfb1-49c2-a517-087f645e4dec/manager/0.log"
Nov 03 13:46:02.190889 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:02.190850    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p8b4c_5b0996d1-8bb0-499c-b88e-b334e4deb0f6/manager/0.log"
Nov 03 13:46:02.681300 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:02.681260    4447 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:46:02.681617 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:02.681322    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:46:02.811193 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:02.811145    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc865c477-6g9s8_72d09d90-d90c-4afe-bd2f-c95980f250e8/manager/1.log"
Nov 03 13:46:02.998050 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:02.997961    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6zhqm_e77dd41f-1fc2-4b1e-be96-902b2186b80e/server/0.log"
Nov 03 13:46:03.174753 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.174688    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-945dt_9a2d78c7-0ebe-45dd-9683-02279bbfff20/manager/0.log"
Nov 03 13:46:03.376799 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.376777    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2trjw_63ddba5e-73fe-472c-bff2-e98dafa5977a/proxy/0.log"
Nov 03 13:46:03.410522 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:46:03.410490    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:46:03.603041 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.603018    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:46:03.730293 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730260    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-data\") pod \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") "
Nov 03 13:46:03.730473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730332    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-envoy\") pod \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") "
Nov 03 13:46:03.730473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730392    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istiod-ca-cert\") pod \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") "
Nov 03 13:46:03.730473 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730423    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-socket\") pod \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") "
Nov 03 13:46:03.730666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730496    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-token\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-token\") pod \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") "
Nov 03 13:46:03.730666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730530    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-kube-api-access-ht5hf\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-kube-api-access-ht5hf\") pod \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") "
Nov 03 13:46:03.730666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730569    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-podinfo\") pod \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") "
Nov 03 13:46:03.730666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730633    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-certs\") pod \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") "
Nov 03 13:46:03.730666 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730663    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-credential-socket\") pod \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\" (UID: \"d3ef6bc9-f218-46fa-9f9a-34a623181d30\") "
Nov 03 13:46:03.730936 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730564    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-data" pod "d3ef6bc9-f218-46fa-9f9a-34a623181d30" (UID: "d3ef6bc9-f218-46fa-9f9a-34a623181d30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:46:03.730936 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730783    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-socket" pod "d3ef6bc9-f218-46fa-9f9a-34a623181d30" (UID: "d3ef6bc9-f218-46fa-9f9a-34a623181d30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:46:03.731065 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.730945    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-certs" pod "d3ef6bc9-f218-46fa-9f9a-34a623181d30" (UID: "d3ef6bc9-f218-46fa-9f9a-34a623181d30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:46:03.731141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.731110    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:03.731141 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.731135    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-data\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:03.731243 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.731150    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-workload-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:03.731312 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.731289    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-credential-socket" pod "d3ef6bc9-f218-46fa-9f9a-34a623181d30" (UID: "d3ef6bc9-f218-46fa-9f9a-34a623181d30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:46:03.731455 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.731306    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istiod-ca-cert" pod "d3ef6bc9-f218-46fa-9f9a-34a623181d30" (UID: "d3ef6bc9-f218-46fa-9f9a-34a623181d30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:46:03.732705 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.732681    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-podinfo" pod "d3ef6bc9-f218-46fa-9f9a-34a623181d30" (UID: "d3ef6bc9-f218-46fa-9f9a-34a623181d30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:46:03.732831 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.732732    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-envoy" pod "d3ef6bc9-f218-46fa-9f9a-34a623181d30" (UID: "d3ef6bc9-f218-46fa-9f9a-34a623181d30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:46:03.732831 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.732803    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-token" pod "d3ef6bc9-f218-46fa-9f9a-34a623181d30" (UID: "d3ef6bc9-f218-46fa-9f9a-34a623181d30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:46:03.732930 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.732907    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-kube-api-access-ht5hf" pod "d3ef6bc9-f218-46fa-9f9a-34a623181d30" (UID: "d3ef6bc9-f218-46fa-9f9a-34a623181d30"). InnerVolumeSpecName "kube-api-access-ht5hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:46:03.760115 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.760086    4447 generic.go:356] "Generic (PLEG): container finished" podID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerID="be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545" exitCode=0
Nov 03 13:46:03.760216 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.760132    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"
Nov 03 13:46:03.760216 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.760170    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" event={"ID":"d3ef6bc9-f218-46fa-9f9a-34a623181d30","Type":"ContainerDied","Data":"be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545"}
Nov 03 13:46:03.760292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.760247    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4" event={"ID":"d3ef6bc9-f218-46fa-9f9a-34a623181d30","Type":"ContainerDied","Data":"ad233b0f69b6d18b3dcf43110a2863d5e95796328c17162371768c75c382eeef"}
Nov 03 13:46:03.760292 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.760268    4447 scope.go:122] "RemoveContainer" containerID="be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545"
Nov 03 13:46:03.772837 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.772819    4447 scope.go:122] "RemoveContainer" containerID="be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545"
Nov 03 13:46:03.773154 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:46:03.773129    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545\": container with ID starting with be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545 not found: ID does not exist" containerID="be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545"
Nov 03 13:46:03.773210 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.773163    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545"} err="failed to get container status \"be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545\": rpc error: code = NotFound desc = could not find container \"be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545\": container with ID starting with be77a63d0dc25812dadbafc15d95be45c3b8001071931a55f9df3a79fad35545 not found: ID does not exist"
Nov 03 13:46:03.786790 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.786763    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"]
Nov 03 13:46:03.793158 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.793135    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-jdfb4"]
Nov 03 13:46:03.832381 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.832337    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-token\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:03.832677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.832379    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht5hf\" (UniqueName: \"kubernetes.io/projected/d3ef6bc9-f218-46fa-9f9a-34a623181d30-kube-api-access-ht5hf\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:03.832677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.832396    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-podinfo\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:03.832677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.832405    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-credential-socket\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:03.832677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.832414    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istio-envoy\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:03.832677 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.832422    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ef6bc9-f218-46fa-9f9a-34a623181d30-istiod-ca-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:03.969265 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:03.969234    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc865c477-6g9s8_72d09d90-d90c-4afe-bd2f-c95980f250e8/manager/0.log"
Nov 03 13:46:04.318977 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:04.318935    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" path="/var/lib/kubelet/pods/d3ef6bc9-f218-46fa-9f9a-34a623181d30/volumes"
Nov 03 13:46:08.370422 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.370391    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc865c477-6g9s8"]
Nov 03 13:46:08.370815 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.370580    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bc865c477-6g9s8" podUID="72d09d90-d90c-4afe-bd2f-c95980f250e8" containerName="manager" containerID="cri-o://b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf" gracePeriod=30
Nov 03 13:46:08.623157 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.623096    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:46:08.779432 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.779394    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-metrics-certs\") pod \"72d09d90-d90c-4afe-bd2f-c95980f250e8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") "
Nov 03 13:46:08.779635 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.779496    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d09d90-d90c-4afe-bd2f-c95980f250e8-kube-api-access-r2hbq\" (UniqueName: \"kubernetes.io/projected/72d09d90-d90c-4afe-bd2f-c95980f250e8-kube-api-access-r2hbq\") pod \"72d09d90-d90c-4afe-bd2f-c95980f250e8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") "
Nov 03 13:46:08.779635 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.779565    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-webhook-cert\") pod \"72d09d90-d90c-4afe-bd2f-c95980f250e8\" (UID: \"72d09d90-d90c-4afe-bd2f-c95980f250e8\") "
Nov 03 13:46:08.781745 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.781715    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-metrics-certs" pod "72d09d90-d90c-4afe-bd2f-c95980f250e8" (UID: "72d09d90-d90c-4afe-bd2f-c95980f250e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:46:08.781859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.781743    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-webhook-cert" pod "72d09d90-d90c-4afe-bd2f-c95980f250e8" (UID: "72d09d90-d90c-4afe-bd2f-c95980f250e8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:46:08.781859 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.781769    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d09d90-d90c-4afe-bd2f-c95980f250e8-kube-api-access-r2hbq" pod "72d09d90-d90c-4afe-bd2f-c95980f250e8" (UID: "72d09d90-d90c-4afe-bd2f-c95980f250e8"). InnerVolumeSpecName "kube-api-access-r2hbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:46:08.789102 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.789068    4447 generic.go:356] "Generic (PLEG): container finished" podID="72d09d90-d90c-4afe-bd2f-c95980f250e8" containerID="b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf" exitCode=0
Nov 03 13:46:08.789214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.789142    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-6g9s8"
Nov 03 13:46:08.789214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.789148    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-6g9s8" event={"ID":"72d09d90-d90c-4afe-bd2f-c95980f250e8","Type":"ContainerDied","Data":"b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf"}
Nov 03 13:46:08.789214 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.789204    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-6g9s8" event={"ID":"72d09d90-d90c-4afe-bd2f-c95980f250e8","Type":"ContainerDied","Data":"1e10d62de8e21d2db3ad46d726f7e54a28eb870c7237069f527363ba65c11ae9"}
Nov 03 13:46:08.789337 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.789228    4447 scope.go:122] "RemoveContainer" containerID="b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf"
Nov 03 13:46:08.805745 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.805725    4447 scope.go:122] "RemoveContainer" containerID="0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67"
Nov 03 13:46:08.813919 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.813870    4447 scope.go:122] "RemoveContainer" containerID="b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf"
Nov 03 13:46:08.814205 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:46:08.814179    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf\": container with ID starting with b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf not found: ID does not exist" containerID="b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf"
Nov 03 13:46:08.814334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.814208    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf"} err="failed to get container status \"b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf\": rpc error: code = NotFound desc = could not find container \"b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf\": container with ID starting with b21b28d3d1ec2ed8048e2898626417124d9fd16b34c1b5a039225972323808bf not found: ID does not exist"
Nov 03 13:46:08.814334 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.814226    4447 scope.go:122] "RemoveContainer" containerID="0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67"
Nov 03 13:46:08.814562 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:46:08.814538    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67\": container with ID starting with 0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67 not found: ID does not exist" containerID="0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67"
Nov 03 13:46:08.814630 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.814573    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67"} err="failed to get container status \"0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67\": rpc error: code = NotFound desc = could not find container \"0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67\": container with ID starting with 0c29017a8890af0b7dcfd78376b39607ce8fe54ec94322a4747a2adf0db91c67 not found: ID does not exist"
Nov 03 13:46:08.816333 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.816314    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc865c477-6g9s8"]
Nov 03 13:46:08.820120 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.820098    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bc865c477-6g9s8"]
Nov 03 13:46:08.880961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.880891    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2hbq\" (UniqueName: \"kubernetes.io/projected/72d09d90-d90c-4afe-bd2f-c95980f250e8-kube-api-access-r2hbq\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:08.880961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.880923    4447 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-webhook-cert\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:08.880961 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:08.880936    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d09d90-d90c-4afe-bd2f-c95980f250e8-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:10.333371 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:46:10.330594    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d09d90-d90c-4afe-bd2f-c95980f250e8" path="/var/lib/kubelet/pods/72d09d90-d90c-4afe-bd2f-c95980f250e8/volumes"
Nov 03 13:46:33.410622 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:46:33.410577    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:47:03.410103 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:47:03.410065    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:47:33.410869 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:47:33.410824    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:48:03.410564 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:48:03.410525    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:48:33.410673 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:48:33.410632    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:49:03.478306 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:49:03.478261    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:49:33.410604 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:49:33.410565    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:50:03.411294 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:50:03.411255    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:50:05.021411 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:50:05.021373    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"]
Nov 03 13:50:05.023662 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:50:05.023630    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c224e916-ng6wz"]
Nov 03 13:50:06.320216 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:50:06.320176    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f740758-1855-4142-8a32-66d78965577c" path="/var/lib/kubelet/pods/3f740758-1855-4142-8a32-66d78965577c/volumes"
Nov 03 13:50:07.127505 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:50:07.127465    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:50:07.127682 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:50:07.127534    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:50:33.410591 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:50:33.410553    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:51:03.409928 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:51:03.409885    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:51:33.411096 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:51:33.411016    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:51:33.669313 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:51:33.669233    4447 scope.go:122] "RemoveContainer" containerID="31ecebba842fa08c7694ab2be202c3dace76f2e607cd8982aea44d041251c76f"
Nov 03 13:52:03.410718 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:52:03.410673    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:52:33.410551 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:52:33.410512    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:52:37.125550 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:52:37.125491    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:52:37.126108 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:52:37.125588    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:53:03.425829 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:53:03.425782    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:53:33.410485 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:53:33.410445    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:54:03.412226 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:54:03.412183    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:54:06.013367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:54:06.012491    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-mnk7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:54:06.013367 ip-10-0-132-167 kubenswrapper[4447]: I1103 13:54:06.012566    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-mnk7k" podUID="6c35106c-e68e-4d61-ad65-88b96270b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:54:33.411361 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:54:33.411247    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:55:03.410749 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:55:03.410710    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:55:33.410936 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:55:33.410897    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:56:03.410013 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:56:03.409973    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:56:33.410500 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:56:33.410462    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:57:03.411575 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:57:03.411532    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:57:33.410132 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:57:33.410052    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:58:03.410277 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:58:03.410239    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:58:33.411623 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:58:33.411575    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:59:03.410379 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:59:03.410310    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 13:59:33.411373 ip-10-0-132-167 kubenswrapper[4447]: E1103 13:59:33.411313    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:00:03.411200 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:00:03.411155    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:00:33.411105 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:00:33.411019    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:01:03.410805 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:01:03.410763    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:01:33.431600 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:01:33.431540    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:02:03.410264 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:02:03.410224    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:02:33.419627 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:02:33.419575    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:03:03.410169 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:03:03.410133    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:03:33.411221 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:03:33.411140    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:03:52.127488 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:03:52.127441    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:03:52.127975 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:03:52.127503    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:04:03.410804 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:04:03.410695    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:04:33.411109 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:04:33.411066    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:05:03.413047 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:05:03.413003    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:05:33.411839 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:05:33.411800    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:06:03.410373 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:06:03.410309    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:06:33.411641 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:06:33.411555    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:07:03.409903 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:07:03.409866    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:07:22.131564 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:22.131514    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-sl776 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:07:22.132128 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:22.131592    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-sl776" podUID="53934dd9-9712-44e9-883b-2693539dc602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:07:33.410693 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:07:33.410653    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:07:55.271471 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:55.271430    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"]
Nov 03 14:07:55.271925 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:55.271702    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx" podUID="e448c30a-a1f7-4d25-9a95-0005c1587096" containerName="maas-api" containerID="cri-o://46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac" gracePeriod=30
Nov 03 14:07:56.518639 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.518613    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 14:07:56.594367 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.594242    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e448c30a-a1f7-4d25-9a95-0005c1587096-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e448c30a-a1f7-4d25-9a95-0005c1587096-openshift-service-ca\") pod \"e448c30a-a1f7-4d25-9a95-0005c1587096\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") "
Nov 03 14:07:56.594367 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.594277    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e448c30a-a1f7-4d25-9a95-0005c1587096-kube-api-access-2d5vg\" (UniqueName: \"kubernetes.io/projected/e448c30a-a1f7-4d25-9a95-0005c1587096-kube-api-access-2d5vg\") pod \"e448c30a-a1f7-4d25-9a95-0005c1587096\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") "
Nov 03 14:07:56.594619 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.594428    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-maas-api-tls\") pod \"e448c30a-a1f7-4d25-9a95-0005c1587096\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") "
Nov 03 14:07:56.594619 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.594464    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-metrics-certs\") pod \"e448c30a-a1f7-4d25-9a95-0005c1587096\" (UID: \"e448c30a-a1f7-4d25-9a95-0005c1587096\") "
Nov 03 14:07:56.594753 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.594649    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e448c30a-a1f7-4d25-9a95-0005c1587096-openshift-service-ca" pod "e448c30a-a1f7-4d25-9a95-0005c1587096" (UID: "e448c30a-a1f7-4d25-9a95-0005c1587096"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:07:56.594885 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.594815    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e448c30a-a1f7-4d25-9a95-0005c1587096-openshift-service-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:07:56.596616 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.596587    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e448c30a-a1f7-4d25-9a95-0005c1587096-kube-api-access-2d5vg" pod "e448c30a-a1f7-4d25-9a95-0005c1587096" (UID: "e448c30a-a1f7-4d25-9a95-0005c1587096"). InnerVolumeSpecName "kube-api-access-2d5vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:07:56.596753 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.596594    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-metrics-certs" pod "e448c30a-a1f7-4d25-9a95-0005c1587096" (UID: "e448c30a-a1f7-4d25-9a95-0005c1587096"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:07:56.596753 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.596616    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-maas-api-tls" pod "e448c30a-a1f7-4d25-9a95-0005c1587096" (UID: "e448c30a-a1f7-4d25-9a95-0005c1587096"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:07:56.644557 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.644522    4447 generic.go:356] "Generic (PLEG): container finished" podID="e448c30a-a1f7-4d25-9a95-0005c1587096" containerID="46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac" exitCode=0
Nov 03 14:07:56.644766 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.644567    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx" event={"ID":"e448c30a-a1f7-4d25-9a95-0005c1587096","Type":"ContainerDied","Data":"46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac"}
Nov 03 14:07:56.644766 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.644604    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx" event={"ID":"e448c30a-a1f7-4d25-9a95-0005c1587096","Type":"ContainerDied","Data":"bbf21a65560cbcd0566b777ea809ab5b1b3848af8ab61359d1de0068d5805cff"}
Nov 03 14:07:56.644766 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.644623    4447 scope.go:122] "RemoveContainer" containerID="46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac"
Nov 03 14:07:56.644766 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.644569    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"
Nov 03 14:07:56.656457 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.656436    4447 scope.go:122] "RemoveContainer" containerID="46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac"
Nov 03 14:07:56.656751 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:07:56.656727    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac\": container with ID starting with 46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac not found: ID does not exist" containerID="46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac"
Nov 03 14:07:56.656825 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.656762    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac"} err="failed to get container status \"46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac\": rpc error: code = NotFound desc = could not find container \"46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac\": container with ID starting with 46702b61151072f8759fd32ebf5e6905eb36b238f79edf200c0b500028f669ac not found: ID does not exist"
Nov 03 14:07:56.669027 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.669004    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"]
Nov 03 14:07:56.673233 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.673208    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-s6sxx"]
Nov 03 14:07:56.695960 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.695933    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-maas-api-tls\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:07:56.695960 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.695958    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e448c30a-a1f7-4d25-9a95-0005c1587096-metrics-certs\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:07:56.696109 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:56.695969    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d5vg\" (UniqueName: \"kubernetes.io/projected/e448c30a-a1f7-4d25-9a95-0005c1587096-kube-api-access-2d5vg\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:07:58.318027 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:07:58.317993    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e448c30a-a1f7-4d25-9a95-0005c1587096" path="/var/lib/kubelet/pods/e448c30a-a1f7-4d25-9a95-0005c1587096/volumes"
Nov 03 14:08:03.410663 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:08:03.410622    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:08:33.411455 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:08:33.411413    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:09:03.411120 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:09:03.411012    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:09:33.410979 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:09:33.410935    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:10:03.415009 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:10:03.414969    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:10:33.411933 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:10:33.411891    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:11:03.411329 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:11:03.411289    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:11:33.411184 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:11:33.411144    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:11:49.884092 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884059    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"]
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884487    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f56fb04c-d9fe-48e9-b361-2a93fb587f71" containerName="payload-pre-processing"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884500    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56fb04c-d9fe-48e9-b361-2a93fb587f71" containerName="payload-pre-processing"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884512    4447 cpu_manager.go:414] "containerMap: removing container" podUID="3f740758-1855-4142-8a32-66d78965577c" containerName="revoke-keys"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884518    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f740758-1855-4142-8a32-66d78965577c" containerName="revoke-keys"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884528    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerName="cleanup"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884533    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerName="cleanup"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884544    4447 cpu_manager.go:414] "containerMap: removing container" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884549    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884557    4447 cpu_manager.go:414] "containerMap: removing container" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerName="istio-proxy"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884562    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerName="istio-proxy"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884569    4447 cpu_manager.go:414] "containerMap: removing container" podUID="72d09d90-d90c-4afe-bd2f-c95980f250e8" containerName="manager"
Nov 03 14:11:49.884573 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884573    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d09d90-d90c-4afe-bd2f-c95980f250e8" containerName="manager"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884582    4447 cpu_manager.go:414] "containerMap: removing container" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884588    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884595    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884599    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884604    4447 cpu_manager.go:414] "containerMap: removing container" podUID="72d09d90-d90c-4afe-bd2f-c95980f250e8" containerName="manager"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884609    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d09d90-d90c-4afe-bd2f-c95980f250e8" containerName="manager"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884621    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e448c30a-a1f7-4d25-9a95-0005c1587096" containerName="maas-api"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884625    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e448c30a-a1f7-4d25-9a95-0005c1587096" containerName="maas-api"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884636    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerName="cleanup"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884641    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerName="cleanup"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884706    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerName="cleanup"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884713    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56fb04c-d9fe-48e9-b361-2a93fb587f71" containerName="payload-pre-processing"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884719    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f740758-1855-4142-8a32-66d78965577c" containerName="revoke-keys"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884724    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerName="cleanup"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884730    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f16b044-d645-494f-a857-00264f18ce26" containerName="istio-proxy"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884736    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="92849677-985f-4a8f-b812-bd24d20b7c8a" containerName="istio-proxy"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884741    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d09d90-d90c-4afe-bd2f-c95980f250e8" containerName="manager"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884746    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaaf70db-dc8c-451f-a584-77b665a67fd3" containerName="istio-proxy"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884751    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ef6bc9-f218-46fa-9f9a-34a623181d30" containerName="istio-proxy"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884757    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d09d90-d90c-4afe-bd2f-c95980f250e8" containerName="manager"
Nov 03 14:11:49.885003 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.884764    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e448c30a-a1f7-4d25-9a95-0005c1587096" containerName="maas-api"
Nov 03 14:11:49.887888 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.887872    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:49.890855 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.890832    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:11:49.890970 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.890873    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:11:49.901955 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.901932    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"]
Nov 03 14:11:49.954147 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.954106    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mm2\" (UniqueName: \"kubernetes.io/projected/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-kube-api-access-r4mm2\") pod \"maas-api-revoke-sub-fd0775b0-zql98\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:49.954147 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.954149    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd0775b0-zql98\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:49.954409 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:49.954238    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd0775b0-zql98\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:50.055504 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.055464    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mm2\" (UniqueName: \"kubernetes.io/projected/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-kube-api-access-r4mm2\") pod \"maas-api-revoke-sub-fd0775b0-zql98\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:50.055504 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.055506    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd0775b0-zql98\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:50.055781 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.055525    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd0775b0-zql98\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:50.056218 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.056191    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd0775b0-zql98\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:50.057911 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.057891    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd0775b0-zql98\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:50.062805 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.062781    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mm2\" (UniqueName: \"kubernetes.io/projected/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-kube-api-access-r4mm2\") pod \"maas-api-revoke-sub-fd0775b0-zql98\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:50.200501 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.200470    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:50.475942 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.475915    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"]
Nov 03 14:11:50.478077 ip-10-0-132-167 kubenswrapper[4447]: W1103 14:11:50.478046    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0ef9a9_7a4f_4890_9fd4_dc038a7a250e.slice/crio-643ac11d0bbc17df030f350ba279259cfaa0d0fdd49a1231a0b0af8cb94964eb WatchSource:0}: Error finding container 643ac11d0bbc17df030f350ba279259cfaa0d0fdd49a1231a0b0af8cb94964eb: Status 404 returned error can't find the container with id 643ac11d0bbc17df030f350ba279259cfaa0d0fdd49a1231a0b0af8cb94964eb
Nov 03 14:11:50.879916 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.879877    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98" event={"ID":"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e","Type":"ContainerStarted","Data":"79992507ae4d6a1f17b4980d96d8cbc4b6ec1b7f2883cef215fa785f84b669da"}
Nov 03 14:11:50.880114 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.879936    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98" event={"ID":"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e","Type":"ContainerStarted","Data":"643ac11d0bbc17df030f350ba279259cfaa0d0fdd49a1231a0b0af8cb94964eb"}
Nov 03 14:11:50.896969 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:50.896914    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98" podStartSLOduration=1.89690114 podStartE2EDuration="1.89690114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:50.895213349 +0000 UTC m=+3798.706881548" watchObservedRunningTime="2026-11-03 14:11:50.89690114 +0000 UTC m=+3798.708569328"
Nov 03 14:11:51.886974 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:51.886941    4447 generic.go:356] "Generic (PLEG): container finished" podID="1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e" containerID="79992507ae4d6a1f17b4980d96d8cbc4b6ec1b7f2883cef215fa785f84b669da" exitCode=0
Nov 03 14:11:51.887161 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:51.886979    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98" event={"ID":"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e","Type":"ContainerDied","Data":"79992507ae4d6a1f17b4980d96d8cbc4b6ec1b7f2883cef215fa785f84b669da"}
Nov 03 14:11:53.029875 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.029847    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:53.084854 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.084821    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-xks-ca\") pod \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") "
Nov 03 14:11:53.085027 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.084948    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-kube-api-access-r4mm2\" (UniqueName: \"kubernetes.io/projected/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-kube-api-access-r4mm2\") pod \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") "
Nov 03 14:11:53.085027 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.085002    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-openshift-ca\") pod \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\" (UID: \"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e\") "
Nov 03 14:11:53.085415 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.085393    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-openshift-ca" pod "1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e" (UID: "1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:11:53.086982 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.086961    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-xks-ca" pod "1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e" (UID: "1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:11:53.086982 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.086974    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-kube-api-access-r4mm2" pod "1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e" (UID: "1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e"). InnerVolumeSpecName "kube-api-access-r4mm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:11:53.186313 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.186285    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-openshift-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:11:53.186313 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.186314    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-maas-api-xks-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:11:53.186525 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.186323    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4mm2\" (UniqueName: \"kubernetes.io/projected/1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e-kube-api-access-r4mm2\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:11:53.897769 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.897739    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98"
Nov 03 14:11:53.897949 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.897772    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0775b0-zql98" event={"ID":"1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e","Type":"ContainerDied","Data":"643ac11d0bbc17df030f350ba279259cfaa0d0fdd49a1231a0b0af8cb94964eb"}
Nov 03 14:11:53.897949 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:11:53.897819    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643ac11d0bbc17df030f350ba279259cfaa0d0fdd49a1231a0b0af8cb94964eb"
Nov 03 14:12:00.141704 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.141662    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"]
Nov 03 14:12:00.142280 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.142259    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e" containerName="revoke-keys"
Nov 03 14:12:00.142400 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.142282    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e" containerName="revoke-keys"
Nov 03 14:12:00.142400 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.142307    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerName="cleanup"
Nov 03 14:12:00.142400 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.142315    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerName="cleanup"
Nov 03 14:12:00.142567 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.142499    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e" containerName="revoke-keys"
Nov 03 14:12:00.142567 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.142516    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98f16b0-a1ca-4d2e-8f57-bd5822efba24" containerName="cleanup"
Nov 03 14:12:00.145972 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.145950    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.157358 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.157282    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:12:00.157478 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.157421    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:12:00.168958 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.168928    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"]
Nov 03 14:12:00.257072 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.257033    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73582c2d-krrhh\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.257269 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.257081    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73582c2d-krrhh\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.257269 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.257148    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57qzg\" (UniqueName: \"kubernetes.io/projected/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-kube-api-access-57qzg\") pod \"maas-api-revoke-sub-73582c2d-krrhh\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.358574 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.358534    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73582c2d-krrhh\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.358758 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.358590    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73582c2d-krrhh\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.358758 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.358653    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57qzg\" (UniqueName: \"kubernetes.io/projected/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-kube-api-access-57qzg\") pod \"maas-api-revoke-sub-73582c2d-krrhh\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.359264 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.359240    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73582c2d-krrhh\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.360868 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.360850    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73582c2d-krrhh\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.366789 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.366764    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57qzg\" (UniqueName: \"kubernetes.io/projected/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-kube-api-access-57qzg\") pod \"maas-api-revoke-sub-73582c2d-krrhh\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.460133 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.460091    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:00.740063 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.740038    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"]
Nov 03 14:12:00.742361 ip-10-0-132-167 kubenswrapper[4447]: W1103 14:12:00.742321    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cb012e_6e8f_4f0e_bc47_0bf1f770244a.slice/crio-5962268737d75183d897839f25f9014882a2cad8a724ae20d85856c24c9fb6d8 WatchSource:0}: Error finding container 5962268737d75183d897839f25f9014882a2cad8a724ae20d85856c24c9fb6d8: Status 404 returned error can't find the container with id 5962268737d75183d897839f25f9014882a2cad8a724ae20d85856c24c9fb6d8
Nov 03 14:12:00.934828 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:00.934791    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh" event={"ID":"30cb012e-6e8f-4f0e-bc47-0bf1f770244a","Type":"ContainerStarted","Data":"5962268737d75183d897839f25f9014882a2cad8a724ae20d85856c24c9fb6d8"}
Nov 03 14:12:01.941426 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:01.941394    4447 generic.go:356] "Generic (PLEG): container finished" podID="30cb012e-6e8f-4f0e-bc47-0bf1f770244a" containerID="eea68f38e985b467d29eb868c277598f714889ee5574f4bda9f65086493ddb25" exitCode=0
Nov 03 14:12:01.941810 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:01.941494    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh" event={"ID":"30cb012e-6e8f-4f0e-bc47-0bf1f770244a","Type":"ContainerDied","Data":"eea68f38e985b467d29eb868c277598f714889ee5574f4bda9f65086493ddb25"}
Nov 03 14:12:03.092913 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.092885    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:03.186328 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.186292    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-xks-ca\") pod \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") "
Nov 03 14:12:03.186505 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.186396    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-kube-api-access-57qzg\" (UniqueName: \"kubernetes.io/projected/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-kube-api-access-57qzg\") pod \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") "
Nov 03 14:12:03.186505 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.186477    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-openshift-ca\") pod \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\" (UID: \"30cb012e-6e8f-4f0e-bc47-0bf1f770244a\") "
Nov 03 14:12:03.187601 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.187559    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-openshift-ca" pod "30cb012e-6e8f-4f0e-bc47-0bf1f770244a" (UID: "30cb012e-6e8f-4f0e-bc47-0bf1f770244a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:03.189112 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.187886    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-openshift-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:12:03.197020 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.196950    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-xks-ca" pod "30cb012e-6e8f-4f0e-bc47-0bf1f770244a" (UID: "30cb012e-6e8f-4f0e-bc47-0bf1f770244a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:03.197334 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.197298    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-kube-api-access-57qzg" pod "30cb012e-6e8f-4f0e-bc47-0bf1f770244a" (UID: "30cb012e-6e8f-4f0e-bc47-0bf1f770244a"). InnerVolumeSpecName "kube-api-access-57qzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:12:03.288709 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.288669    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-maas-api-xks-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:12:03.288709 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.288706    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57qzg\" (UniqueName: \"kubernetes.io/projected/30cb012e-6e8f-4f0e-bc47-0bf1f770244a-kube-api-access-57qzg\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:12:03.410069 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:12:03.410025    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:12:03.953157 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.953103    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh" event={"ID":"30cb012e-6e8f-4f0e-bc47-0bf1f770244a","Type":"ContainerDied","Data":"5962268737d75183d897839f25f9014882a2cad8a724ae20d85856c24c9fb6d8"}
Nov 03 14:12:03.953157 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.953149    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73582c2d-krrhh"
Nov 03 14:12:03.953157 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:03.953165    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5962268737d75183d897839f25f9014882a2cad8a724ae20d85856c24c9fb6d8"
Nov 03 14:12:22.064501 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.064418    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"]
Nov 03 14:12:22.065002 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.064862    4447 cpu_manager.go:414] "containerMap: removing container" podUID="30cb012e-6e8f-4f0e-bc47-0bf1f770244a" containerName="revoke-keys"
Nov 03 14:12:22.065002 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.064872    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30cb012e-6e8f-4f0e-bc47-0bf1f770244a" containerName="revoke-keys"
Nov 03 14:12:22.065002 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.064938    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="30cb012e-6e8f-4f0e-bc47-0bf1f770244a" containerName="revoke-keys"
Nov 03 14:12:22.068076 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.068059    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.073683 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.073657    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:12:22.073930 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.073763    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:12:22.077223 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.077198    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"]
Nov 03 14:12:22.179919 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.179878    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrb9d\" (UniqueName: \"kubernetes.io/projected/c5c9180a-2630-4705-aba7-22741fd9ba42-kube-api-access-zrb9d\") pod \"maas-api-revoke-sub-637c3f5d-zvc7b\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.180090 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.180048    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637c3f5d-zvc7b\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.180153 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.180101    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637c3f5d-zvc7b\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.280617 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.280574    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637c3f5d-zvc7b\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.280804 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.280643    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637c3f5d-zvc7b\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.280804 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.280683    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrb9d\" (UniqueName: \"kubernetes.io/projected/c5c9180a-2630-4705-aba7-22741fd9ba42-kube-api-access-zrb9d\") pod \"maas-api-revoke-sub-637c3f5d-zvc7b\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.281290 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.281266    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637c3f5d-zvc7b\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.283162 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.283133    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637c3f5d-zvc7b\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.290070 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.290047    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrb9d\" (UniqueName: \"kubernetes.io/projected/c5c9180a-2630-4705-aba7-22741fd9ba42-kube-api-access-zrb9d\") pod \"maas-api-revoke-sub-637c3f5d-zvc7b\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.382720 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.382617    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:22.670204 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:22.670111    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"]
Nov 03 14:12:22.674857 ip-10-0-132-167 kubenswrapper[4447]: W1103 14:12:22.674820    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c9180a_2630_4705_aba7_22741fd9ba42.slice/crio-8372db282b58a4a606e964f7c8803d5701715fd09a4ac4fbb3d740a35dfbad4e WatchSource:0}: Error finding container 8372db282b58a4a606e964f7c8803d5701715fd09a4ac4fbb3d740a35dfbad4e: Status 404 returned error can't find the container with id 8372db282b58a4a606e964f7c8803d5701715fd09a4ac4fbb3d740a35dfbad4e
Nov 03 14:12:23.061063 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:23.061024    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b" event={"ID":"c5c9180a-2630-4705-aba7-22741fd9ba42","Type":"ContainerStarted","Data":"f6df4a0a27cb979aec99397516e9f286992df716bafcf5a9ff33ab519269dc02"}
Nov 03 14:12:23.061239 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:23.061085    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b" event={"ID":"c5c9180a-2630-4705-aba7-22741fd9ba42","Type":"ContainerStarted","Data":"8372db282b58a4a606e964f7c8803d5701715fd09a4ac4fbb3d740a35dfbad4e"}
Nov 03 14:12:23.077360 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:23.077297    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b" podStartSLOduration=1.077282254 podStartE2EDuration="1.077282254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:23.075266586 +0000 UTC m=+3830.886934782" watchObservedRunningTime="2026-11-03 14:12:23.077282254 +0000 UTC m=+3830.888950441"
Nov 03 14:12:24.068076 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:24.068039    4447 generic.go:356] "Generic (PLEG): container finished" podID="c5c9180a-2630-4705-aba7-22741fd9ba42" containerID="f6df4a0a27cb979aec99397516e9f286992df716bafcf5a9ff33ab519269dc02" exitCode=0
Nov 03 14:12:24.068286 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:24.068129    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b" event={"ID":"c5c9180a-2630-4705-aba7-22741fd9ba42","Type":"ContainerDied","Data":"f6df4a0a27cb979aec99397516e9f286992df716bafcf5a9ff33ab519269dc02"}
Nov 03 14:12:25.210419 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.210393    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:25.311820 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.311781    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c9180a-2630-4705-aba7-22741fd9ba42-kube-api-access-zrb9d\" (UniqueName: \"kubernetes.io/projected/c5c9180a-2630-4705-aba7-22741fd9ba42-kube-api-access-zrb9d\") pod \"c5c9180a-2630-4705-aba7-22741fd9ba42\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") "
Nov 03 14:12:25.311998 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.311854    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-openshift-ca\") pod \"c5c9180a-2630-4705-aba7-22741fd9ba42\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") "
Nov 03 14:12:25.311998 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.311946    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-xks-ca\") pod \"c5c9180a-2630-4705-aba7-22741fd9ba42\" (UID: \"c5c9180a-2630-4705-aba7-22741fd9ba42\") "
Nov 03 14:12:25.312317 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.312288    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-openshift-ca" pod "c5c9180a-2630-4705-aba7-22741fd9ba42" (UID: "c5c9180a-2630-4705-aba7-22741fd9ba42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:25.314018 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.313989    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-xks-ca" pod "c5c9180a-2630-4705-aba7-22741fd9ba42" (UID: "c5c9180a-2630-4705-aba7-22741fd9ba42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:25.314113 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.314011    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c9180a-2630-4705-aba7-22741fd9ba42-kube-api-access-zrb9d" pod "c5c9180a-2630-4705-aba7-22741fd9ba42" (UID: "c5c9180a-2630-4705-aba7-22741fd9ba42"). InnerVolumeSpecName "kube-api-access-zrb9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:12:25.413050 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.412953    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-xks-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:12:25.413050 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.412992    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrb9d\" (UniqueName: \"kubernetes.io/projected/c5c9180a-2630-4705-aba7-22741fd9ba42-kube-api-access-zrb9d\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:12:25.413050 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:25.413007    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9180a-2630-4705-aba7-22741fd9ba42-maas-api-openshift-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:12:26.080766 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:26.080727    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b" event={"ID":"c5c9180a-2630-4705-aba7-22741fd9ba42","Type":"ContainerDied","Data":"8372db282b58a4a606e964f7c8803d5701715fd09a4ac4fbb3d740a35dfbad4e"}
Nov 03 14:12:26.080766 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:26.080751    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637c3f5d-zvc7b"
Nov 03 14:12:26.080982 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:26.080786    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8372db282b58a4a606e964f7c8803d5701715fd09a4ac4fbb3d740a35dfbad4e"
Nov 03 14:12:33.414263 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:12:33.414218    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:12:48.318374 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.318314    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"]
Nov 03 14:12:48.318936 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.318915    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c5c9180a-2630-4705-aba7-22741fd9ba42" containerName="revoke-keys"
Nov 03 14:12:48.319031 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.318938    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c9180a-2630-4705-aba7-22741fd9ba42" containerName="revoke-keys"
Nov 03 14:12:48.319103 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.319087    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c9180a-2630-4705-aba7-22741fd9ba42" containerName="revoke-keys"
Nov 03 14:12:48.322329 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.322307    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.326989 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.326958    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:12:48.327125 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.327065    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:12:48.330742 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.330719    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"]
Nov 03 14:12:48.453176 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.453132    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2304cc4b-kg5pb\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.453176 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.453188    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2304cc4b-kg5pb\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.453512 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.453401    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2f9\" (UniqueName: \"kubernetes.io/projected/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-kube-api-access-fp2f9\") pod \"maas-api-revoke-sub-2304cc4b-kg5pb\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.554596 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.554554    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2f9\" (UniqueName: \"kubernetes.io/projected/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-kube-api-access-fp2f9\") pod \"maas-api-revoke-sub-2304cc4b-kg5pb\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.554793 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.554623    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2304cc4b-kg5pb\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.554793 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.554651    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2304cc4b-kg5pb\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.555394 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.555370    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2304cc4b-kg5pb\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.556941 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.556918    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2304cc4b-kg5pb\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.562645 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.562625    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2f9\" (UniqueName: \"kubernetes.io/projected/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-kube-api-access-fp2f9\") pod \"maas-api-revoke-sub-2304cc4b-kg5pb\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.638336 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.638248    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:48.916137 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:48.916030    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"]
Nov 03 14:12:48.919007 ip-10-0-132-167 kubenswrapper[4447]: W1103 14:12:48.918976    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcde5ea_dd50_4f7d_bd8d_e5539fb5dd83.slice/crio-3bbb22687f3bf9c416bcc09be2f54081d4a2525e35b27fb346dd4771e3da652f WatchSource:0}: Error finding container 3bbb22687f3bf9c416bcc09be2f54081d4a2525e35b27fb346dd4771e3da652f: Status 404 returned error can't find the container with id 3bbb22687f3bf9c416bcc09be2f54081d4a2525e35b27fb346dd4771e3da652f
Nov 03 14:12:49.258674 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:49.258629    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb" event={"ID":"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83","Type":"ContainerStarted","Data":"2f51d12def0ee26bb25446b6c538f4f0aa7d87e00173a8cf0d28ae32f5ced119"}
Nov 03 14:12:49.258828 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:49.258718    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb" event={"ID":"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83","Type":"ContainerStarted","Data":"3bbb22687f3bf9c416bcc09be2f54081d4a2525e35b27fb346dd4771e3da652f"}
Nov 03 14:12:49.274769 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:49.274717    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb" podStartSLOduration=1.274701669 podStartE2EDuration="1.274701669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:49.273578626 +0000 UTC m=+3857.085246816" watchObservedRunningTime="2026-11-03 14:12:49.274701669 +0000 UTC m=+3857.086369858"
Nov 03 14:12:50.265360 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:50.265315    4447 generic.go:356] "Generic (PLEG): container finished" podID="7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83" containerID="2f51d12def0ee26bb25446b6c538f4f0aa7d87e00173a8cf0d28ae32f5ced119" exitCode=0
Nov 03 14:12:50.265746 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:50.265407    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb" event={"ID":"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83","Type":"ContainerDied","Data":"2f51d12def0ee26bb25446b6c538f4f0aa7d87e00173a8cf0d28ae32f5ced119"}
Nov 03 14:12:51.406160 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.406135    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:51.484819 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.484784    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-kube-api-access-fp2f9\" (UniqueName: \"kubernetes.io/projected/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-kube-api-access-fp2f9\") pod \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") "
Nov 03 14:12:51.485013 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.484857    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-xks-ca\") pod \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") "
Nov 03 14:12:51.485013 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.484893    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-openshift-ca\") pod \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\" (UID: \"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83\") "
Nov 03 14:12:51.485288 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.485265    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-openshift-ca" pod "7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83" (UID: "7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:51.486979 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.486947    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-xks-ca" pod "7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83" (UID: "7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:51.487062 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.487010    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-kube-api-access-fp2f9" pod "7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83" (UID: "7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83"). InnerVolumeSpecName "kube-api-access-fp2f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:12:51.586009 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.585918    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2f9\" (UniqueName: \"kubernetes.io/projected/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-kube-api-access-fp2f9\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:12:51.586009 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.585954    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-xks-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:12:51.586009 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:51.585964    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83-maas-api-openshift-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:12:52.277306 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:52.277271    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb" event={"ID":"7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83","Type":"ContainerDied","Data":"3bbb22687f3bf9c416bcc09be2f54081d4a2525e35b27fb346dd4771e3da652f"}
Nov 03 14:12:52.277306 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:52.277295    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304cc4b-kg5pb"
Nov 03 14:12:52.277575 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:52.277318    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bbb22687f3bf9c416bcc09be2f54081d4a2525e35b27fb346dd4771e3da652f"
Nov 03 14:12:58.534906 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.534871    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"]
Nov 03 14:12:58.535434 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.535312    4447 cpu_manager.go:414] "containerMap: removing container" podUID="7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83" containerName="revoke-keys"
Nov 03 14:12:58.535434 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.535327    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83" containerName="revoke-keys"
Nov 03 14:12:58.535526 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.535441    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83" containerName="revoke-keys"
Nov 03 14:12:58.538819 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.538802    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.542720 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.542692    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:12:58.543291 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.543268    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:12:58.545883 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.545858    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"]
Nov 03 14:12:58.649626 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.649589    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b2d46e5-msdkv\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.649823 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.649646    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b2d46e5-msdkv\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.649823 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.649725    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmpm5\" (UniqueName: \"kubernetes.io/projected/13fcd267-6f08-4d3a-b101-bd032f9171d6-kube-api-access-vmpm5\") pod \"maas-api-revoke-sub-3b2d46e5-msdkv\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.750812 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.750777    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b2d46e5-msdkv\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.750812 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.750816    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmpm5\" (UniqueName: \"kubernetes.io/projected/13fcd267-6f08-4d3a-b101-bd032f9171d6-kube-api-access-vmpm5\") pod \"maas-api-revoke-sub-3b2d46e5-msdkv\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.751046 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.750889    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b2d46e5-msdkv\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.751489 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.751468    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b2d46e5-msdkv\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.753257 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.753238    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b2d46e5-msdkv\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.758739 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.758719    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmpm5\" (UniqueName: \"kubernetes.io/projected/13fcd267-6f08-4d3a-b101-bd032f9171d6-kube-api-access-vmpm5\") pod \"maas-api-revoke-sub-3b2d46e5-msdkv\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:58.854569 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:58.854473    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:12:59.129651 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:59.129625    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"]
Nov 03 14:12:59.131388 ip-10-0-132-167 kubenswrapper[4447]: W1103 14:12:59.131306    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fcd267_6f08_4d3a_b101_bd032f9171d6.slice/crio-f2db56f4e45733794e9a26f7b691aa066fe35c468a138347fd52e2f019dac15a WatchSource:0}: Error finding container f2db56f4e45733794e9a26f7b691aa066fe35c468a138347fd52e2f019dac15a: Status 404 returned error can't find the container with id f2db56f4e45733794e9a26f7b691aa066fe35c468a138347fd52e2f019dac15a
Nov 03 14:12:59.317329 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:12:59.317294    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv" event={"ID":"13fcd267-6f08-4d3a-b101-bd032f9171d6","Type":"ContainerStarted","Data":"f2db56f4e45733794e9a26f7b691aa066fe35c468a138347fd52e2f019dac15a"}
Nov 03 14:13:00.323575 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:00.323540    4447 generic.go:356] "Generic (PLEG): container finished" podID="13fcd267-6f08-4d3a-b101-bd032f9171d6" containerID="c5dc0dfb80f8d6fa2e3790d3725c7f61a5685e88f66a50a056e486d0673f3776" exitCode=0
Nov 03 14:13:00.323986 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:00.323620    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv" event={"ID":"13fcd267-6f08-4d3a-b101-bd032f9171d6","Type":"ContainerDied","Data":"c5dc0dfb80f8d6fa2e3790d3725c7f61a5685e88f66a50a056e486d0673f3776"}
Nov 03 14:13:01.473953 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.473922    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:13:01.581524 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.581407    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fcd267-6f08-4d3a-b101-bd032f9171d6-kube-api-access-vmpm5\" (UniqueName: \"kubernetes.io/projected/13fcd267-6f08-4d3a-b101-bd032f9171d6-kube-api-access-vmpm5\") pod \"13fcd267-6f08-4d3a-b101-bd032f9171d6\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") "
Nov 03 14:13:01.581524 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.581512    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-xks-ca\") pod \"13fcd267-6f08-4d3a-b101-bd032f9171d6\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") "
Nov 03 14:13:01.581774 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.581617    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-openshift-ca\") pod \"13fcd267-6f08-4d3a-b101-bd032f9171d6\" (UID: \"13fcd267-6f08-4d3a-b101-bd032f9171d6\") "
Nov 03 14:13:01.582153 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.582078    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-openshift-ca" pod "13fcd267-6f08-4d3a-b101-bd032f9171d6" (UID: "13fcd267-6f08-4d3a-b101-bd032f9171d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:13:01.584163 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.584137    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-xks-ca" pod "13fcd267-6f08-4d3a-b101-bd032f9171d6" (UID: "13fcd267-6f08-4d3a-b101-bd032f9171d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:13:01.584249 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.584201    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fcd267-6f08-4d3a-b101-bd032f9171d6-kube-api-access-vmpm5" pod "13fcd267-6f08-4d3a-b101-bd032f9171d6" (UID: "13fcd267-6f08-4d3a-b101-bd032f9171d6"). InnerVolumeSpecName "kube-api-access-vmpm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:13:01.683454 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.683396    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmpm5\" (UniqueName: \"kubernetes.io/projected/13fcd267-6f08-4d3a-b101-bd032f9171d6-kube-api-access-vmpm5\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:13:01.683454 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.683447    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-xks-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:13:01.683454 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:01.683458    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fcd267-6f08-4d3a-b101-bd032f9171d6-maas-api-openshift-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:13:02.335077 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:02.335056    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv"
Nov 03 14:13:02.335219 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:02.335090    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2d46e5-msdkv" event={"ID":"13fcd267-6f08-4d3a-b101-bd032f9171d6","Type":"ContainerDied","Data":"f2db56f4e45733794e9a26f7b691aa066fe35c468a138347fd52e2f019dac15a"}
Nov 03 14:13:02.335219 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:02.335134    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2db56f4e45733794e9a26f7b691aa066fe35c468a138347fd52e2f019dac15a"
Nov 03 14:13:03.410944 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:13:03.410903    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:13:30.837787 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.837744    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"]
Nov 03 14:13:30.838590 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.838570    4447 cpu_manager.go:414] "containerMap: removing container" podUID="13fcd267-6f08-4d3a-b101-bd032f9171d6" containerName="revoke-keys"
Nov 03 14:13:30.838654 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.838594    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fcd267-6f08-4d3a-b101-bd032f9171d6" containerName="revoke-keys"
Nov 03 14:13:30.838764 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.838751    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fcd267-6f08-4d3a-b101-bd032f9171d6" containerName="revoke-keys"
Nov 03 14:13:30.842626 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.842606    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:30.846393 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.846228    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:13:30.846570 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.846543    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:13:30.849454 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.849412    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"]
Nov 03 14:13:30.871320 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.871286    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6759bc2-9lwjk\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:30.871539 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.871337    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwmw\" (UniqueName: \"kubernetes.io/projected/5ac32e92-2444-4410-90d8-daca49e488ad-kube-api-access-5wwmw\") pod \"maas-api-revoke-sub-d6759bc2-9lwjk\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:30.871539 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.871463    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6759bc2-9lwjk\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:30.972281 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.972241    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6759bc2-9lwjk\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:30.972483 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.972407    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6759bc2-9lwjk\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:30.972483 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.972463    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwmw\" (UniqueName: \"kubernetes.io/projected/5ac32e92-2444-4410-90d8-daca49e488ad-kube-api-access-5wwmw\") pod \"maas-api-revoke-sub-d6759bc2-9lwjk\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:30.973114 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.973090    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6759bc2-9lwjk\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:30.975235 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.975215    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6759bc2-9lwjk\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:30.980921 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:30.980894    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwmw\" (UniqueName: \"kubernetes.io/projected/5ac32e92-2444-4410-90d8-daca49e488ad-kube-api-access-5wwmw\") pod \"maas-api-revoke-sub-d6759bc2-9lwjk\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:31.159844 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:31.159753    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:31.439353 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:31.439312    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"]
Nov 03 14:13:31.441279 ip-10-0-132-167 kubenswrapper[4447]: W1103 14:13:31.441249    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac32e92_2444_4410_90d8_daca49e488ad.slice/crio-84841e9870fdb9d748020726b9a415501f51bcafab94596dc9299af58b28f938 WatchSource:0}: Error finding container 84841e9870fdb9d748020726b9a415501f51bcafab94596dc9299af58b28f938: Status 404 returned error can't find the container with id 84841e9870fdb9d748020726b9a415501f51bcafab94596dc9299af58b28f938
Nov 03 14:13:31.543576 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:31.543544    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk" event={"ID":"5ac32e92-2444-4410-90d8-daca49e488ad","Type":"ContainerStarted","Data":"84841e9870fdb9d748020726b9a415501f51bcafab94596dc9299af58b28f938"}
Nov 03 14:13:32.550802 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:32.550714    4447 generic.go:356] "Generic (PLEG): container finished" podID="5ac32e92-2444-4410-90d8-daca49e488ad" containerID="72b20ec3a505e9cc440027da6328a2bbe922ce23584ed33e51140fb9e30812df" exitCode=0
Nov 03 14:13:32.551198 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:32.550803    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk" event={"ID":"5ac32e92-2444-4410-90d8-daca49e488ad","Type":"ContainerDied","Data":"72b20ec3a505e9cc440027da6328a2bbe922ce23584ed33e51140fb9e30812df"}
Nov 03 14:13:33.411612 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:13:33.411569    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:13:33.723181 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.723152    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:33.799363 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.799305    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac32e92-2444-4410-90d8-daca49e488ad-kube-api-access-5wwmw\" (UniqueName: \"kubernetes.io/projected/5ac32e92-2444-4410-90d8-daca49e488ad-kube-api-access-5wwmw\") pod \"5ac32e92-2444-4410-90d8-daca49e488ad\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") "
Nov 03 14:13:33.799534 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.799374    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-xks-ca\") pod \"5ac32e92-2444-4410-90d8-daca49e488ad\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") "
Nov 03 14:13:33.799534 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.799409    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-openshift-ca\") pod \"5ac32e92-2444-4410-90d8-daca49e488ad\" (UID: \"5ac32e92-2444-4410-90d8-daca49e488ad\") "
Nov 03 14:13:33.799824 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.799802    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-openshift-ca" pod "5ac32e92-2444-4410-90d8-daca49e488ad" (UID: "5ac32e92-2444-4410-90d8-daca49e488ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:13:33.801485 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.801463    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-xks-ca" pod "5ac32e92-2444-4410-90d8-daca49e488ad" (UID: "5ac32e92-2444-4410-90d8-daca49e488ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:13:33.801594 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.801527    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac32e92-2444-4410-90d8-daca49e488ad-kube-api-access-5wwmw" pod "5ac32e92-2444-4410-90d8-daca49e488ad" (UID: "5ac32e92-2444-4410-90d8-daca49e488ad"). InnerVolumeSpecName "kube-api-access-5wwmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:13:33.900157 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.900126    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-openshift-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:13:33.900157 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.900152    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wwmw\" (UniqueName: \"kubernetes.io/projected/5ac32e92-2444-4410-90d8-daca49e488ad-kube-api-access-5wwmw\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:13:33.900157 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:33.900163    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac32e92-2444-4410-90d8-daca49e488ad-maas-api-xks-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:13:34.562496 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:34.562460    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk"
Nov 03 14:13:34.562671 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:34.562463    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6759bc2-9lwjk" event={"ID":"5ac32e92-2444-4410-90d8-daca49e488ad","Type":"ContainerDied","Data":"84841e9870fdb9d748020726b9a415501f51bcafab94596dc9299af58b28f938"}
Nov 03 14:13:34.562671 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:34.562582    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84841e9870fdb9d748020726b9a415501f51bcafab94596dc9299af58b28f938"
Nov 03 14:13:41.091039 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.091001    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"]
Nov 03 14:13:41.091610 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.091488    4447 cpu_manager.go:414] "containerMap: removing container" podUID="5ac32e92-2444-4410-90d8-daca49e488ad" containerName="revoke-keys"
Nov 03 14:13:41.091610 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.091501    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac32e92-2444-4410-90d8-daca49e488ad" containerName="revoke-keys"
Nov 03 14:13:41.091610 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.091580    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac32e92-2444-4410-90d8-daca49e488ad" containerName="revoke-keys"
Nov 03 14:13:41.094931 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.094906    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.103488 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.103462    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:13:41.103615 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.103565    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:13:41.106424 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.106399    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"]
Nov 03 14:13:41.175426 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.175389    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45a292b2-j5tc4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.175621 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.175446    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45a292b2-j5tc4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.175621 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.175568    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h568z\" (UniqueName: \"kubernetes.io/projected/8fa2235c-197c-4be3-9882-1511e2df12e4-kube-api-access-h568z\") pod \"maas-api-revoke-sub-45a292b2-j5tc4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.277109 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.277078    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h568z\" (UniqueName: \"kubernetes.io/projected/8fa2235c-197c-4be3-9882-1511e2df12e4-kube-api-access-h568z\") pod \"maas-api-revoke-sub-45a292b2-j5tc4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.277316 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.277174    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45a292b2-j5tc4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.277316 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.277207    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45a292b2-j5tc4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.277882 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.277856    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45a292b2-j5tc4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.279493 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.279474    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45a292b2-j5tc4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.284979 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.284956    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h568z\" (UniqueName: \"kubernetes.io/projected/8fa2235c-197c-4be3-9882-1511e2df12e4-kube-api-access-h568z\") pod \"maas-api-revoke-sub-45a292b2-j5tc4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.410568 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.410495    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:41.691887 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:41.691858    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"]
Nov 03 14:13:41.693882 ip-10-0-132-167 kubenswrapper[4447]: W1103 14:13:41.693855    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa2235c_197c_4be3_9882_1511e2df12e4.slice/crio-2e39d272c4c3aadc2bf32b4357f1a112eb18e7bb320b6cadc8cc9a2916e4191b WatchSource:0}: Error finding container 2e39d272c4c3aadc2bf32b4357f1a112eb18e7bb320b6cadc8cc9a2916e4191b: Status 404 returned error can't find the container with id 2e39d272c4c3aadc2bf32b4357f1a112eb18e7bb320b6cadc8cc9a2916e4191b
Nov 03 14:13:42.610974 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:42.610926    4447 generic.go:356] "Generic (PLEG): container finished" podID="8fa2235c-197c-4be3-9882-1511e2df12e4" containerID="43d0d78dd41006c995afaa33bc1b4ebdb88571385ecfdd4423254351aa8cb175" exitCode=0
Nov 03 14:13:42.611518 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:42.611055    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4" event={"ID":"8fa2235c-197c-4be3-9882-1511e2df12e4","Type":"ContainerDied","Data":"43d0d78dd41006c995afaa33bc1b4ebdb88571385ecfdd4423254351aa8cb175"}
Nov 03 14:13:42.611518 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:42.611113    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4" event={"ID":"8fa2235c-197c-4be3-9882-1511e2df12e4","Type":"ContainerStarted","Data":"2e39d272c4c3aadc2bf32b4357f1a112eb18e7bb320b6cadc8cc9a2916e4191b"}
Nov 03 14:13:43.914124 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:43.914095    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:44.016326 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.016276    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-openshift-ca\") pod \"8fa2235c-197c-4be3-9882-1511e2df12e4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") "
Nov 03 14:13:44.016563 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.016374    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-xks-ca\") pod \"8fa2235c-197c-4be3-9882-1511e2df12e4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") "
Nov 03 14:13:44.016563 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.016452    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa2235c-197c-4be3-9882-1511e2df12e4-kube-api-access-h568z\" (UniqueName: \"kubernetes.io/projected/8fa2235c-197c-4be3-9882-1511e2df12e4-kube-api-access-h568z\") pod \"8fa2235c-197c-4be3-9882-1511e2df12e4\" (UID: \"8fa2235c-197c-4be3-9882-1511e2df12e4\") "
Nov 03 14:13:44.017881 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.017852    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-openshift-ca" pod "8fa2235c-197c-4be3-9882-1511e2df12e4" (UID: "8fa2235c-197c-4be3-9882-1511e2df12e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:13:44.020409 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.020377    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-xks-ca" pod "8fa2235c-197c-4be3-9882-1511e2df12e4" (UID: "8fa2235c-197c-4be3-9882-1511e2df12e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:13:44.022574 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.022539    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa2235c-197c-4be3-9882-1511e2df12e4-kube-api-access-h568z" pod "8fa2235c-197c-4be3-9882-1511e2df12e4" (UID: "8fa2235c-197c-4be3-9882-1511e2df12e4"). InnerVolumeSpecName "kube-api-access-h568z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:13:44.118985 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.117457    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-openshift-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:13:44.118985 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.117492    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa2235c-197c-4be3-9882-1511e2df12e4-maas-api-xks-ca\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:13:44.118985 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.117510    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h568z\" (UniqueName: \"kubernetes.io/projected/8fa2235c-197c-4be3-9882-1511e2df12e4-kube-api-access-h568z\") on node \"ip-10-0-132-167.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:13:44.625775 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.625674    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4" event={"ID":"8fa2235c-197c-4be3-9882-1511e2df12e4","Type":"ContainerDied","Data":"2e39d272c4c3aadc2bf32b4357f1a112eb18e7bb320b6cadc8cc9a2916e4191b"}
Nov 03 14:13:44.625775 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.625709    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a292b2-j5tc4"
Nov 03 14:13:44.625775 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:13:44.625734    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e39d272c4c3aadc2bf32b4357f1a112eb18e7bb320b6cadc8cc9a2916e4191b"
Nov 03 14:14:03.410287 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:14:03.410249    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:14:11.101595 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:11.101564    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-tbkh6_d5388bcf-9195-4a03-89b5-d50dcb20b89d/manager/0.log"
Nov 03 14:14:11.277506 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:11.277470    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8w2nb_1ab4053b-cfb1-49c2-a517-087f645e4dec/manager/0.log"
Nov 03 14:14:11.473919 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:11.473886    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p8b4c_5b0996d1-8bb0-499c-b88e-b334e4deb0f6/manager/0.log"
Nov 03 14:14:11.949928 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:11.949903    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6zhqm_e77dd41f-1fc2-4b1e-be96-902b2186b80e/server/0.log"
Nov 03 14:14:12.132496 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:12.132464    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-945dt_9a2d78c7-0ebe-45dd-9683-02279bbfff20/manager/0.log"
Nov 03 14:14:12.335313 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:12.335239    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2trjw_63ddba5e-73fe-472c-bff2-e98dafa5977a/proxy/0.log"
Nov 03 14:14:13.622366 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:13.622316    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2304cc4b-kg5pb_7dcde5ea-dd50-4f7d-bd8d-e5539fb5dd83/revoke-keys/0.log"
Nov 03 14:14:13.789090 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:13.789057    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3b2d46e5-msdkv_13fcd267-6f08-4d3a-b101-bd032f9171d6/revoke-keys/0.log"
Nov 03 14:14:13.956162 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:13.956132    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-45a292b2-j5tc4_8fa2235c-197c-4be3-9882-1511e2df12e4/revoke-keys/0.log"
Nov 03 14:14:14.123111 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:14.123067    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-637c3f5d-zvc7b_c5c9180a-2630-4705-aba7-22741fd9ba42/revoke-keys/0.log"
Nov 03 14:14:14.290999 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:14.290919    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-73582c2d-krrhh_30cb012e-6e8f-4f0e-bc47-0bf1f770244a/revoke-keys/0.log"
Nov 03 14:14:14.456072 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:14.456032    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d6759bc2-9lwjk_5ac32e92-2444-4410-90d8-daca49e488ad/revoke-keys/0.log"
Nov 03 14:14:14.629184 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:14.629101    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fd0775b0-zql98_1f0ef9a9-7a4f-4890-9fd4-dc038a7a250e/revoke-keys/0.log"
Nov 03 14:14:14.796982 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:14.796953    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-fjs8c_3d7ec807-b74e-4dfe-a48d-a1531d7ec2c4/postgres/0.log"
Nov 03 14:14:18.999808 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:18.999768    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hqpn5_fc8f96e5-8722-4503-8361-f5ade2d145f3/discovery/0.log"
Nov 03 14:14:19.366138 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:19.366061    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d8c65c8f-ltmbm_b0d73ec9-8ac7-4179-86ed-51c93ddc0017/kube-auth-proxy/0.log"
Nov 03 14:14:20.124745 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:20.124714    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-mnk7k_6c35106c-e68e-4d61-ad65-88b96270b873/istio-proxy/0.log"
Nov 03 14:14:20.308135 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:20.308103    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-sl776_53934dd9-9712-44e9-883b-2693539dc602/istio-proxy/0.log"
Nov 03 14:14:21.062289 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:21.062246    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-z544g_08586908-beb4-471e-8071-0ffc382d6ac2/istio-proxy/0.log"
Nov 03 14:14:33.411722 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:14:33.411681    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:14:38.646196 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:38.646162    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9wrnl_0ef0c534-57ad-4269-8d12-e2a5879a3eed/global-pull-secret-syncer/0.log"
Nov 03 14:14:39.036758 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:39.036724    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d9722_2aaa0065-2805-432d-8c2e-ef03591b6c90/konnectivity-agent/0.log"
Nov 03 14:14:39.110958 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:39.110922    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-167.us-east-2.compute.internal_6c4c57a57b1c5fc6c67e60988666a273/haproxy/0.log"
Nov 03 14:14:49.500726 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.500693    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395945ec-ad7b-446f-a8d9-87b611ba8523/alertmanager/0.log"
Nov 03 14:14:49.536624 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.536599    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395945ec-ad7b-446f-a8d9-87b611ba8523/config-reloader/0.log"
Nov 03 14:14:49.568747 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.568720    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395945ec-ad7b-446f-a8d9-87b611ba8523/kube-rbac-proxy-web/0.log"
Nov 03 14:14:49.602630 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.602599    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395945ec-ad7b-446f-a8d9-87b611ba8523/kube-rbac-proxy/0.log"
Nov 03 14:14:49.636417 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.636388    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395945ec-ad7b-446f-a8d9-87b611ba8523/kube-rbac-proxy-metric/0.log"
Nov 03 14:14:49.668496 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.668469    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395945ec-ad7b-446f-a8d9-87b611ba8523/prom-label-proxy/0.log"
Nov 03 14:14:49.703433 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.703399    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395945ec-ad7b-446f-a8d9-87b611ba8523/init-config-reloader/0.log"
Nov 03 14:14:49.821280 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.821198    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bl5ld_e439b615-5ffe-4341-b2d7-de50d3ff1eeb/kube-state-metrics/0.log"
Nov 03 14:14:49.845043 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.845014    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bl5ld_e439b615-5ffe-4341-b2d7-de50d3ff1eeb/kube-rbac-proxy-main/0.log"
Nov 03 14:14:49.876328 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.876302    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bl5ld_e439b615-5ffe-4341-b2d7-de50d3ff1eeb/kube-rbac-proxy-self/0.log"
Nov 03 14:14:49.941540 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.941507    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"]
Nov 03 14:14:49.942086 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.942062    4447 cpu_manager.go:414] "containerMap: removing container" podUID="8fa2235c-197c-4be3-9882-1511e2df12e4" containerName="revoke-keys"
Nov 03 14:14:49.942086 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.942083    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa2235c-197c-4be3-9882-1511e2df12e4" containerName="revoke-keys"
Nov 03 14:14:49.942282 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.942152    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa2235c-197c-4be3-9882-1511e2df12e4" containerName="revoke-keys"
Nov 03 14:14:49.945437 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.945423    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:49.951059 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.951033    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vtpt6\"/\"kube-root-ca.crt\""
Nov 03 14:14:49.953206 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.953179    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vtpt6\"/\"openshift-service-ca.crt\""
Nov 03 14:14:49.953662 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.953301    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vtpt6\"/\"default-dockercfg-vkdhx\""
Nov 03 14:14:49.954397 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:49.954373    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"]
Nov 03 14:14:50.047369 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.047316    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-sys\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.047576 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.047455    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-lib-modules\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.047576 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.047502    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-proc\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.047709 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.047640    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-podres\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.047766 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.047710    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfwm\" (UniqueName: \"kubernetes.io/projected/d7105637-47bd-4f92-bc06-6674037378ab-kube-api-access-dvfwm\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.148769 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.148672    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfwm\" (UniqueName: \"kubernetes.io/projected/d7105637-47bd-4f92-bc06-6674037378ab-kube-api-access-dvfwm\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.148769 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.148728    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-sys\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.148769 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.148763    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-lib-modules\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.149045 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.148783    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-proc\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.149045 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.148839    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-podres\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.149045 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.148882    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-sys\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.149045 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.148899    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-lib-modules\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.149045 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.148894    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-proc\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.149045 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.148936    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7105637-47bd-4f92-bc06-6674037378ab-podres\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.155912 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.155880    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfwm\" (UniqueName: \"kubernetes.io/projected/d7105637-47bd-4f92-bc06-6674037378ab-kube-api-access-dvfwm\") pod \"perf-node-gather-daemonset-wdzs8\" (UID: \"d7105637-47bd-4f92-bc06-6674037378ab\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.237186 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.237144    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zjkhw_bdb25057-8a32-4c8a-829a-a3b8b0c1487c/node-exporter/0.log"
Nov 03 14:14:50.260588 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.260558    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.283864 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.283839    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zjkhw_bdb25057-8a32-4c8a-829a-a3b8b0c1487c/kube-rbac-proxy/0.log"
Nov 03 14:14:50.318050 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.318024    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zjkhw_bdb25057-8a32-4c8a-829a-a3b8b0c1487c/init-textfile/0.log"
Nov 03 14:14:50.354072 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.354037    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2vs99_0c682e8f-5f71-4a30-95f7-c86622eff068/kube-rbac-proxy-main/0.log"
Nov 03 14:14:50.386770 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.386743    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2vs99_0c682e8f-5f71-4a30-95f7-c86622eff068/kube-rbac-proxy-self/0.log"
Nov 03 14:14:50.423468 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.423390    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2vs99_0c682e8f-5f71-4a30-95f7-c86622eff068/openshift-state-metrics/0.log"
Nov 03 14:14:50.541111 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.541076    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"]
Nov 03 14:14:50.543931 ip-10-0-132-167 kubenswrapper[4447]: W1103 14:14:50.543899    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd7105637_47bd_4f92_bc06_6674037378ab.slice/crio-b9aa211637f6d931f0c58e4eead2a5d00bb90a002b479908c1b46883b6f3e639 WatchSource:0}: Error finding container b9aa211637f6d931f0c58e4eead2a5d00bb90a002b479908c1b46883b6f3e639: Status 404 returned error can't find the container with id b9aa211637f6d931f0c58e4eead2a5d00bb90a002b479908c1b46883b6f3e639
Nov 03 14:14:50.750945 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.750913    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-m5zdq_21f68575-8422-4984-ba8d-e4bc90daa416/prometheus-operator/0.log"
Nov 03 14:14:50.776479 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.776444    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-m5zdq_21f68575-8422-4984-ba8d-e4bc90daa416/kube-rbac-proxy/0.log"
Nov 03 14:14:50.809843 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.809812    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lp9rq_6257b998-d021-47de-850b-65d0c5e03eb0/prometheus-operator-admission-webhook/0.log"
Nov 03 14:14:50.960268 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.960231    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686567946f-4frpb_8f575583-222a-45d1-a7cc-b7dbdeae608c/thanos-query/0.log"
Nov 03 14:14:50.989178 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.989142    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8" event={"ID":"d7105637-47bd-4f92-bc06-6674037378ab","Type":"ContainerStarted","Data":"fc3d28ef6a241758c7142328f536dfe1d7ca532b05ed6660a2729f316085d7a3"}
Nov 03 14:14:50.989363 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.989204    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:50.989363 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.989222    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8" event={"ID":"d7105637-47bd-4f92-bc06-6674037378ab","Type":"ContainerStarted","Data":"b9aa211637f6d931f0c58e4eead2a5d00bb90a002b479908c1b46883b6f3e639"}
Nov 03 14:14:50.992086 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:50.992064    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686567946f-4frpb_8f575583-222a-45d1-a7cc-b7dbdeae608c/kube-rbac-proxy-web/0.log"
Nov 03 14:14:51.005105 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:51.004994    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8" podStartSLOduration=2.004981205 podStartE2EDuration="2.004981205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:51.002564853 +0000 UTC m=+3978.814233041" watchObservedRunningTime="2026-11-03 14:14:51.004981205 +0000 UTC m=+3978.816649408"
Nov 03 14:14:51.025069 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:51.025047    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686567946f-4frpb_8f575583-222a-45d1-a7cc-b7dbdeae608c/kube-rbac-proxy/0.log"
Nov 03 14:14:51.059815 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:51.059788    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686567946f-4frpb_8f575583-222a-45d1-a7cc-b7dbdeae608c/prom-label-proxy/0.log"
Nov 03 14:14:51.093981 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:51.093956    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686567946f-4frpb_8f575583-222a-45d1-a7cc-b7dbdeae608c/kube-rbac-proxy-rules/0.log"
Nov 03 14:14:51.130269 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:51.130240    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686567946f-4frpb_8f575583-222a-45d1-a7cc-b7dbdeae608c/kube-rbac-proxy-metrics/0.log"
Nov 03 14:14:52.072822 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:52.072789    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-tbkh6_d5388bcf-9195-4a03-89b5-d50dcb20b89d/manager/0.log"
Nov 03 14:14:52.188154 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:52.188113    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8w2nb_1ab4053b-cfb1-49c2-a517-087f645e4dec/manager/0.log"
Nov 03 14:14:53.072986 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:53.072938    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p8b4c_5b0996d1-8bb0-499c-b88e-b334e4deb0f6/manager/0.log"
Nov 03 14:14:53.249684 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:53.249655    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6zhqm_e77dd41f-1fc2-4b1e-be96-902b2186b80e/server/0.log"
Nov 03 14:14:53.770069 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:53.770035    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-964cc69f9-gxnx5_25c9c616-da13-4317-8b5c-78584a815f02/console/0.log"
Nov 03 14:14:53.842723 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:53.842692    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-j565n_6da37367-7ecb-4213-a856-c7ae36609bae/download-server/0.log"
Nov 03 14:14:55.174159 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:55.174121    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-945dt_9a2d78c7-0ebe-45dd-9683-02279bbfff20/manager/0.log"
Nov 03 14:14:55.198667 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:55.198641    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2trjw_63ddba5e-73fe-472c-bff2-e98dafa5977a/proxy/0.log"
Nov 03 14:14:56.186784 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:56.186720    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v5xss_b4100403-c8f8-4ac1-98c4-2795d5a11b10/dns/0.log"
Nov 03 14:14:56.217227 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:56.217200    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v5xss_b4100403-c8f8-4ac1-98c4-2795d5a11b10/kube-rbac-proxy/0.log"
Nov 03 14:14:56.316681 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:56.316653    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fgqkh_44f5fcad-a6fb-4bd2-99aa-f4973415aafa/dns-node-resolver/0.log"
Nov 03 14:14:56.381552 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:56.381512    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-tbkh6_d5388bcf-9195-4a03-89b5-d50dcb20b89d/manager/0.log"
Nov 03 14:14:56.479170 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:56.479085    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8w2nb_1ab4053b-cfb1-49c2-a517-087f645e4dec/manager/0.log"
Nov 03 14:14:57.008554 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:57.008524    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-wdzs8"
Nov 03 14:14:57.133045 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:57.133010    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5dcf6c45d8-f5t74_39f044b1-cfab-41fb-a72a-819e22c431cf/registry/0.log"
Nov 03 14:14:57.263590 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:57.263496    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-88kk5_ba1ed75b-f38b-4589-87fa-0c8d22abe784/node-ca/0.log"
Nov 03 14:14:57.331108 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:57.331071    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p8b4c_5b0996d1-8bb0-499c-b88e-b334e4deb0f6/manager/0.log"
Nov 03 14:14:57.512418 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:57.512389    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6zhqm_e77dd41f-1fc2-4b1e-be96-902b2186b80e/server/0.log"
Nov 03 14:14:59.662758 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:59.662720    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-945dt_9a2d78c7-0ebe-45dd-9683-02279bbfff20/manager/0.log"
Nov 03 14:14:59.851997 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:14:59.851958    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2trjw_63ddba5e-73fe-472c-bff2-e98dafa5977a/proxy/0.log"
Nov 03 14:15:00.164892 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.164858    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"]
Nov 03 14:15:00.169746 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.169718    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"
Nov 03 14:15:00.174852 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.174824    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:15:00.175466 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.175443    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"]
Nov 03 14:15:00.221724 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.221687    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/197e3406-2e04-4cec-b282-7e7f0a8e2a56-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895255-qpn7s\" (UID: \"197e3406-2e04-4cec-b282-7e7f0a8e2a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"
Nov 03 14:15:00.221920 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.221751    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf5h2\" (UniqueName: \"kubernetes.io/projected/197e3406-2e04-4cec-b282-7e7f0a8e2a56-kube-api-access-hf5h2\") pod \"maas-api-key-cleanup-29895255-qpn7s\" (UID: \"197e3406-2e04-4cec-b282-7e7f0a8e2a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"
Nov 03 14:15:00.322381 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.322326    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/197e3406-2e04-4cec-b282-7e7f0a8e2a56-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895255-qpn7s\" (UID: \"197e3406-2e04-4cec-b282-7e7f0a8e2a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"
Nov 03 14:15:00.322584 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.322407    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf5h2\" (UniqueName: \"kubernetes.io/projected/197e3406-2e04-4cec-b282-7e7f0a8e2a56-kube-api-access-hf5h2\") pod \"maas-api-key-cleanup-29895255-qpn7s\" (UID: \"197e3406-2e04-4cec-b282-7e7f0a8e2a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"
Nov 03 14:15:00.323420 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.323397    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/197e3406-2e04-4cec-b282-7e7f0a8e2a56-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895255-qpn7s\" (UID: \"197e3406-2e04-4cec-b282-7e7f0a8e2a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"
Nov 03 14:15:00.342292 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.342097    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf5h2\" (UniqueName: \"kubernetes.io/projected/197e3406-2e04-4cec-b282-7e7f0a8e2a56-kube-api-access-hf5h2\") pod \"maas-api-key-cleanup-29895255-qpn7s\" (UID: \"197e3406-2e04-4cec-b282-7e7f0a8e2a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"
Nov 03 14:15:00.490000 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.489965    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"
Nov 03 14:15:00.786785 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:00.786755    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s"]
Nov 03 14:15:00.789648 ip-10-0-132-167 kubenswrapper[4447]: W1103 14:15:00.789608    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197e3406_2e04_4cec_b282_7e7f0a8e2a56.slice/crio-5c8d68097b3dc60cb2e957a4f52112e3443aa514b0bfb94c928466e452d5db1b WatchSource:0}: Error finding container 5c8d68097b3dc60cb2e957a4f52112e3443aa514b0bfb94c928466e452d5db1b: Status 404 returned error can't find the container with id 5c8d68097b3dc60cb2e957a4f52112e3443aa514b0bfb94c928466e452d5db1b
Nov 03 14:15:01.063001 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:01.062958    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s" event={"ID":"197e3406-2e04-4cec-b282-7e7f0a8e2a56","Type":"ContainerStarted","Data":"5c8d68097b3dc60cb2e957a4f52112e3443aa514b0bfb94c928466e452d5db1b"}
Nov 03 14:15:01.079100 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:01.079072    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-tbkh6_d5388bcf-9195-4a03-89b5-d50dcb20b89d/manager/0.log"
Nov 03 14:15:01.209984 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:01.209887    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8w2nb_1ab4053b-cfb1-49c2-a517-087f645e4dec/manager/0.log"
Nov 03 14:15:02.071956 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:02.071891    4447 generic.go:356] "Generic (PLEG): container finished" podID="197e3406-2e04-4cec-b282-7e7f0a8e2a56" containerID="5bd6c173d57b5cd0931de7defa59c07001d5086ec11b4c70000249e53224e6e9" exitCode=60
Nov 03 14:15:02.072452 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:02.072221    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s" event={"ID":"197e3406-2e04-4cec-b282-7e7f0a8e2a56","Type":"ContainerDied","Data":"5bd6c173d57b5cd0931de7defa59c07001d5086ec11b4c70000249e53224e6e9"}
Nov 03 14:15:02.072452 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:02.072271    4447 scope.go:122] "RemoveContainer" containerID="5bd6c173d57b5cd0931de7defa59c07001d5086ec11b4c70000249e53224e6e9"
Nov 03 14:15:02.598153 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:02.598064    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p8b4c_5b0996d1-8bb0-499c-b88e-b334e4deb0f6/manager/0.log"
Nov 03 14:15:02.701581 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:02.701535    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hqpn5_fc8f96e5-8722-4503-8361-f5ade2d145f3/discovery/0.log"
Nov 03 14:15:02.785294 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:02.785268    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6zhqm_e77dd41f-1fc2-4b1e-be96-902b2186b80e/server/0.log"
Nov 03 14:15:02.853381 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:02.853281    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d8c65c8f-ltmbm_b0d73ec9-8ac7-4179-86ed-51c93ddc0017/kube-auth-proxy/0.log"
Nov 03 14:15:03.048167 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:03.048130    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-mnk7k_6c35106c-e68e-4d61-ad65-88b96270b873/istio-proxy/0.log"
Nov 03 14:15:03.081385 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:03.081353    4447 generic.go:356] "Generic (PLEG): container finished" podID="197e3406-2e04-4cec-b282-7e7f0a8e2a56" containerID="b38bff0f84a32d253bbb58e3c6d7e5cedbb8065dccdd33571fa518c8fe990c62" exitCode=60
Nov 03 14:15:03.081858 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:03.081381    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s" event={"ID":"197e3406-2e04-4cec-b282-7e7f0a8e2a56","Type":"ContainerDied","Data":"b38bff0f84a32d253bbb58e3c6d7e5cedbb8065dccdd33571fa518c8fe990c62"}
Nov 03 14:15:03.081858 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:03.081464    4447 scope.go:122] "RemoveContainer" containerID="5bd6c173d57b5cd0931de7defa59c07001d5086ec11b4c70000249e53224e6e9"
Nov 03 14:15:03.081858 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:03.081763    4447 scope.go:122] "RemoveContainer" containerID="b38bff0f84a32d253bbb58e3c6d7e5cedbb8065dccdd33571fa518c8fe990c62"
Nov 03 14:15:03.082062 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:15:03.082038    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895255-qpn7s_odh-ai-gateway-infra(197e3406-2e04-4cec-b282-7e7f0a8e2a56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s" podUID="197e3406-2e04-4cec-b282-7e7f0a8e2a56"
Nov 03 14:15:03.096200 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:03.096173    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-sl776_53934dd9-9712-44e9-883b-2693539dc602/istio-proxy/0.log"
Nov 03 14:15:03.264785 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:03.264747    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-z544g_08586908-beb4-471e-8071-0ffc382d6ac2/istio-proxy/0.log"
Nov 03 14:15:03.417807 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:15:03.417761    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqvjq" containerName="extractor"
Nov 03 14:15:04.091403 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:04.091338    4447 scope.go:122] "RemoveContainer" containerID="b38bff0f84a32d253bbb58e3c6d7e5cedbb8065dccdd33571fa518c8fe990c62"
Nov 03 14:15:04.091861 ip-10-0-132-167 kubenswrapper[4447]: E1103 14:15:04.091680    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895255-qpn7s_odh-ai-gateway-infra(197e3406-2e04-4cec-b282-7e7f0a8e2a56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-qpn7s" podUID="197e3406-2e04-4cec-b282-7e7f0a8e2a56"
Nov 03 14:15:05.432414 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:05.432375    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-945dt_9a2d78c7-0ebe-45dd-9683-02279bbfff20/manager/0.log"
Nov 03 14:15:05.477924 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:05.477861    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7hff_e0bc4a29-7ff7-4b22-8f9f-56906b70ee41/serve-healthcheck-canary/0.log"
Nov 03 14:15:05.521479 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:05.521440    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2trjw_63ddba5e-73fe-472c-bff2-e98dafa5977a/proxy/0.log"
Nov 03 14:15:05.896620 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:05.896526    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hqpn5_fc8f96e5-8722-4503-8361-f5ade2d145f3/discovery/0.log"
Nov 03 14:15:05.932040 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:05.932008    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d8c65c8f-ltmbm_b0d73ec9-8ac7-4179-86ed-51c93ddc0017/kube-auth-proxy/0.log"
Nov 03 14:15:06.042257 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:06.042221    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-mnk7k_6c35106c-e68e-4d61-ad65-88b96270b873/istio-proxy/0.log"
Nov 03 14:15:06.067662 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:06.067633    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-sl776_53934dd9-9712-44e9-883b-2693539dc602/istio-proxy/0.log"
Nov 03 14:15:06.169724 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:06.169650    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-z544g_08586908-beb4-471e-8071-0ffc382d6ac2/istio-proxy/0.log"
Nov 03 14:15:06.542971 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:06.542934    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dqvjq_d58899f4-efb6-4d21-b1eb-fcea8c046356/kube-rbac-proxy/0.log"
Nov 03 14:15:06.580557 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:06.580529    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dqvjq_d58899f4-efb6-4d21-b1eb-fcea8c046356/exporter/0.log"
Nov 03 14:15:06.619409 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:06.619339    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dqvjq_d58899f4-efb6-4d21-b1eb-fcea8c046356/extractor/0.log"
Nov 03 14:15:06.785842 ip-10-0-132-167 kubenswrapper[4447]: I1103 14:15:06.785807    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-p694g-sqtcz_f5ef0a05-3328-401e-946c-cc7b1582b15c/insights-gathering/0.log"